Added a template model definition file to the user documentation
Added a section on sharing assets to the user documentation
Added a series of basic walkthroughs of core platform functionality, with examples, to the user documentation
Improvements
Updated the base Docker image used by Jupyter Notebook visualisations to the latest Jupyter Data Science Notebook
Updated the user documentation to recommend using Rancher Desktop for running Docker commands when preparing models
Fixes
Fixed an issue causing deleted assets to appear on groups pages
Fixed an issue causing workflow instances run by other users to be visible
18th February 2025
New
Added support for numeric enumerated model parameters in the workflow builder
Improvements
Empty fields are no longer included in downloaded model definition files
Fixes
Fixed an issue causing an error to display when requesting to add assets to the public group
29th January 2025
New
Added new optional metadata fields [embargo_end_date], [funding], [project_name], [project_url] to datasets, models, and workflows. These new fields are displayed on the asset details pages
Added new optional metadata field [dataset_source] to datasets. This new field will be displayed on the dataset details page
Improvements
Expanded documentation on Docker commands used when creating a DAFNI ready model
Expanded documentation on model definition files to include a more detailed example containing all possible fields
Fixes
Fixed an issue causing loop steps to display incorrectly when viewing workflow instances, and preventing looping workflows from running
13th November 2024
New
Upgraded underlying DAFNI Kubernetes cluster
Improvements
Added a default file path ('outputs/**/*') for publish steps during workflow creation
Replaced the row of action icons in the workflow parameters table with a three-dot 'Actions List' drop-down button
Added a three-dot ‘Actions List' drop-down button to the workflow instances table. This includes a 'View detailed status’ button to navigate to the instance details page, replacing the link on the instance name
Updated the 'Delete Version' button on the dataset details page to display only when there is more than one version of a dataset
Added an 'Apply Changes' button during workflow creation. This button functions as clicking directly onto the canvas, saving updates and returning to the canvas only view
7th October 2024
New
Added a new Parameter Sweep step type to workflows. This new step type allows you to iterate over a model many times within a workflow, with the option of generating parameter values. For more details please see the documentation
Added a new, optional metadata field 'Subject' for models and workflows
Added a dropdown button for model actions (edit/delete) on the model details page, in place of the 'Edit model' button
Added a 'Download model definition' button to the model actions, allowing download of a copy of a model's definition file
Fixes
Fixed an issue where updating a model would fail because of missing metadata fields
24th July 2024
New
When searching the data catalogue, if a search term is used, the results are now ordered by relevance, rather than date
When searching the data catalogue using the search term 'openclim', the OpenCLIM_public_assets dataset will be pinned to the top of the search results. This dataset contains details of all the publicly available OpenCLIM assets on DAFNI
Improvements
Added the ability to delete steps in the Workflow builder using the backspace key
When viewing a Workflow details page, the Parameter Set table now displays the Workflow Version ID with which the Parameter Set was last updated
Improved the layout of the Dataset details page
Fixes
Fixed an issue where users could see Dataset Versions they did not have permission to view in the Version drop-down list
Fixed an issue where users could sometimes not run Sequential Looping Workflows with up to 50 iterations
23rd May 2024
New
BETA: Added the option to send users email notifications when Workflows finish and when Models finish ingesting. This is a beta feature. For more information on using beta features please see the documentation
Improvements
Updated our User Documentation on Model Metadata and added a Getting Started on DAFNI section
Fixes
Fixed an issue where deleted Dataset Versions where showing in the version list
Fixed an issue where deleting the non-latest version of a Dataset would show an error
Fixed an issue where deleting the latest Model/Workflow Version and then attempting to upload a new Version was showing an error
Fixed an issue where viewing a Workflow details page could error if there were no Instances
11th March 2024
New
Added a Delete Instances button in the Visualisation Catalogue
Updated the Privacy Policy in the terms and conditions to allow for sending platform email notifications
Improvements
Decreased the lifetime of PUT urls used for uploading files to DAFNI to 4 hours
Fixes
Fixed an issue where deleted Assets were appearing in the Catalogues
Fixed an issue where deleting a Dataset Version and then attempting to upload a new Version was showing an error
6th February 2024
New
Added a Stop Workflow button on the Workflow Instance page to allow users to stop their own Workflows
Added a duplicate Parameter Set button in the Parameter Set table on the Workflow details page
BETA: Added the option to send users email notifications when Datasets finish ingesting. This is a beta feature. For more information on using beta features please see the documentation
Improvements
Updated our User Documentation to describe how to set Boolean values in yaml files
Updated the Dataset details page to show a clearer error if there is a backend error
Updated the layout of the Dataset details page
Changed the search behaviour in the Data Catalogue to no longer automatically match partial words. This can still be done using an asterisk (e.g. infra*)
Added a check to ensure users only have view or read permissions on assets accessed via the public group
Added a check to ensure default Datasets exist and the current user has access to them when using the Model's default values
An error will now be returned when uploading a new Dataset Version if you do not have update permissions on the latest Version
Fixes
Fixed a bug on the Data Catalogue page where all other filter values show as 0 when one filter is selected
When executing a Workflow, an error will now be shown if the Workflow is trying to use a Model that has not yet finished ingesting
Breaking Changes
Removed the Metadata endpoint from our Search and Discovery service, now replaced by an endpoint in the NID
23rd November 2023
New
Added new metadata fields [contact_point_name], [contact_point_email], [licence], and [rights] to Models and Workflows. The contact point name and email are now required fields, while rights and licence are optional fields. These new fields are displayed on the Model/Workflow details pages; existing assets will show N/A until they are updated by the asset owner. Note: as a result the Model definition schema has been updated, you will need to update any Model definition files you plan to upload to have "api_version" set to v1beta3
Added Dockerisation guide for C++ and f77 models
Released the DAFNI CLI python package, source code for which can be found here
Improvements
Improved efficiency of calls to get catalogue of Data resulting in slightly faster loading of Data catalogue
Improved efficiency of calls to get catalogue of Groups resulting in slightly faster loading of Groups catalogue
Added an activity counter to monitor how often API endpoints are called across the DAFNI services
Spaces are now stripped from the start and end of the creator field in Dataset metadata when DAFNI sends the metadata to the APIs
Fixes
Workflows can no longer be executed if any Datasets in the Parameter Set are not fully ingested, which was previously causing Workflows to fail
Fixed an incorrect spelling for the default creative commons licence url in Dataset metadata to now point to the correct page
Fixed an issue that caused permissions problems in dataset upload process
4th July 2023
New
Reworked how Datasets are uploaded in DAFNI removing the 5GB limit on file sizes. There is still a soft limit of ~50GB for total Dataset size
Added an option to manually specify the Geojson for Dataset spatial coverage when uploading a Dataset and when filling in the metadata in Publish and Publish and Visualise steps in Workflows
Improvements
Improved efficiency of calls to get the steps of an existing Workflow resulting in faster loading of Workflow builder pages (viewing/editing a Workflow or Parameter Set)
Improved efficiency of calls to get catalogue of Data resulting in slightly faster loading of Data catalogue
Errors will now be shown when viewing/editing a Workflow or Parameter Set if you do not have permissions on the Models/Workflows used by the Workflow
Spaces are now stripped from the start and end of fields that take paths when DAFNI sends the paths to the APIs
Fixes
Workflows no longer error when two Datasets used by a Model contain a duplicate file. The duplicate file will have its name appended with an underscore followed by the Dataset version ID
Fixed an issue that caused files in a Dataset to be opened in a browser instead of downloaded when they are downloaded from the Dataset details page
11th April 2023
New
When selecting a Model for a Workflow's Model step you can now change the version of the Model you would like to run in the Model step
When selecting a Workflow for a Workflow's Loop step you can now change the version of the Workflow you would like to loop over
We have added checkboxes to the data file table when viewing a Dataset's details page, these checkboxes can be used to bulk download a Dataset's files. Note: the checkbox in the header of the table will only select all visible files in the table, in order to download all files then change the rows per page of the table to 'All' then check the checkbox in the header of the table to select all files. Also note: files will be downloaded individually, not in an archive
Added a `View workflow` button to the Workflow's details page which will allow you to view a Workflow's steps but not edit their contents - this will be visible to users who have been given permissions to view but not execute your Workflow
Added a button to the Parameter Set's table on the Workflow's details page which will allow you to view a Parameter Set, and the parameters/datasets entered into the Parameter Set
Improvements
Reduced timeout issues when uploading Datasets that have lots of files
Added a character count and character limit to the registration form, to make clear the character limit on the form's fields
Fixes
Fixed an issue that resulted in user's getting a 403 message when requesting to make an asset public on DAFNI. An email will now be sent to support@dafni.ac.uk to let us know that you have requested to make an asset public, at that point we will review the request and approve or deny the request
Fixed an issue where the oldest metadata for a Dataset was being shown in a group's Dataset list rather than the most recent metadata
Fixed an issue where Datasets were only showing in the Data catalogue if you had permissions to see the most recent version of the Dataset
Fixed an issue where optional numeric parameters were being set to zero in Model steps in some situations
Fixed a rare issue where step positions in a Workflow could be non-integer resulting in a confusing error being shown to users
Fixed an issue that prevented Model steps from receiving outputs from Loop steps
Fixed an issue that caused default values not to be loaded into an existing Parameter Set's Model steps when new Model steps were added to a Workflow
29th November 2022
New
When viewing a Workflow Instance's status there are new buttons for publish and visualisation steps `Go to dataset` and `Go to visualisation` clicking these will take you to the dataset/visualisation that was created by that step. Note: Visualisations may take a few minutes to deploy depending on the load on DAFNI at the time, if you click to go to the visualisation on a recently completed visualisation step you may see a `Service not available` error, please try again in a few minutes if that does happen
Added a checkbox to the edit pages for Datasets, Models and Workflows near the Version Message fields. When checked the new version you upload will be shared with the same users and groups that the version you are editing was. The checkbox is checked by default, and only visible to users that have admin permissions on the asset. Note: new versions of assets will still need to be shared publicly on DAFNI as there is still a manual approval process in place for public assets on DAFNI.
We have replaced the front end package we used to enable building of Workflows on DAFNI, this has brought with it several new features and changes to the way Workflows are built on DAFNI
The key for the Workflows builder has now moved to the left hand side of the canvas, this is now interactive - steps can be dragged from the list onto the canvas to build your Workflow
You no longer need to press apply changes when changing a step's configuration. All changes are saved to the Workflow that is being built locally. Note: this does mean that you can no longer easily "revert" changes made to steps
Steps must be connected together with links, these links are a representation of what steps are dependent on what other steps - as a result you can now visually display the branches in your Workflow, these will run in parallel on DAFNI
Steps can now be deleted by selecting a step and pressing the delete key on your keyboard
Steps can be re-positioned in the Workflow, including inserting steps between two existing steps
Fields that require selecting a step will now update if the selected step's name changes
Validation of step fields now happens when entering data into step fields, when deselecting a step and when pressing the continue button to move on to the next step of Workflow creation. This is to enable validation issues to be raised as soon as possible
Added a large tooltip to explain how to use the new workflow builder canvas. Accessed by pressing the ? button next to the `Add workflow steps` heading
Can now zoom the Workflow builder canvas in and out with CTRL + Scrolling
We have reordered Workflow creation so adding/editing steps now comes before entering metadata
We now validate step names on the DAFNI website to prevent usage of reserved step names - "dag", "data-retrieval-step", "data-transfer-step", "directory-structure-creator"
Steps now have IDs automatically set by DAFNI. Users of the API's will need to provide these ids and ensure they are immutable
Rearranged entry of metadata for publish and visualisation steps and added extra help text to highlight the difference between manual metadata entry and metadata from model step
Improvements
Added hint text to data keywords field and loop parameter values field to explain how to enter new values
Fixed an issue with numeric fields that caused decimal points and "unnecessary" zeroes after a decimal from being unexpectedly deleted
Changed secondary buttons to have a white background to improve visibility
Changed sequential loop steps to be blue to signal that parallel and sequential loops perform very similar actions
Drop down fields in DAFNI are now searchable to speed up entering information
We have redesigned the use of the Geonames API to provide bounding boxes for Dataset spatial information to improve usability. Note: searching no longer happens as you type, you will need to either press "enter" on your keyboard or the search button provided to start the search
We now validate the outputs section of the Model definition file, outputs will now need the name, type and description fields. Note: as a result the Model definition schema has been updated, you will need to update any Model definition files you plan to upload to have "api_version" set to v1beta2
Fixes
Fixed an issue where Datasets shared in groups with edit permissions were not able to be updated by group members
Fixed an issue that prevented sorting of loop iteration data when viewing a running loop step's status
Fixed an issue in loops where some system steps were duplicated preventing the loop from starting
Fixed an issue with group permissions where if the same asset was shared with you through two different groups you wouldn't necessarily be given the highest level of permissions provided to you by the groups
25th August 2022
New
Added a search bar to the groups catalogue which allow you to filter the list of groups by the name and description of the group
Added a search bar to the assets tables when adding assets to groups in DAFNI to allow you to filter the list of assets by name
Added a button when adding assets to groups in DAFNI that allows you to change the version of the asset you are adding to the group
Added a button to the Workflow details page to allow for deletion of Parameter Sets
Added a button to the Workflow details page which allows you to duplicate a Workflow. When duplicating a Workflow you can choose the name of the new Workflow, choose whether to copy the Parameter Sets or not and also choose whether to add the new Workflow to the groups the original has been added to
Improvements
Navigating to non-existent pages, pages with non-existent UUIDs or pages with invalid UUIDs will now display an appropriate error
Added pagination to the Data catalogue reducing improving the speed of the initial page load as well as searching the catalogue
Redesigned the process for adding assets to groups in DAFNI
Dialog when deleting a Dataset version now specifies that you are deleting a Dataset version not the overall Dataset
Redesigned the process for creating a Visualisation directly from a Dataset. You will now need to navigate to the Visualisations catalogue and press the "Create visualisation" button, this will then allow you to choose which Datasets you want to add to the Visualisation
Users who have self-registered but have not had their accounts approved yet can now login to DAFNI and see their account details and the status of their account. Once your account has been approved you will be able to navigate around the rest of DAFNI
Improved the error messages when issues are encountered on uploading a Dataset to the Data catalogue
Fixes
Fixed various issues with the validation of Model definitions so that they provide more informative error messages
Fixed an issue where long descriptions of Workflows were not being displayed correctly on the Workflow details page
28th February 2022
New
Added two new kinds of step to Workflows - parallel and sequential loop steps which allow you to loop over Workflows, varying parameters or dataslots
Parallel loop steps will run each iteration of the loop independently
Sequential loop steps will run each iteration one after the other in order and will pass the data between the iterations
Added a new type of Model, Service Models, that will run for the entire length of a Workflow. This allows you to run a database alongside the other Models in your Workflow and to send Data to and from that database. For more information see the documentation
Workflows can now be shared in Groups
Can now select Datasets in the Data catalogue and create a Visualisation directly from those Datasets
Added a tooltip to explain how to glob multiple data files at the same time when creating a publish / visualisation step
Improvements
Can now sort Workflow instances by status on the Workflow details page
Time is now shown alongside version upload dates
Added a tooltip to dataslots on the Model details page to show the dataslot description
Fixes
Fixed an issue that meant the name of a Model was showing when adding a Model to a group rather than the display name
Fixed an issue that prevented optional dataslots from loading on the Model details page
Fixed an issue that prevented users from editing Parameter Sets after adding new steps to a Workflow
Fixed an issue that resulted in a new Model version being added to a Model instead of creating a brand new Model
Adding single spaces to required metadata fields for Workflows and Parameter Sets will now cause an error on the field instead of only failing on upload of the Parameter Set
7th September 2021
New
Added a publisher field to Model and Workflow metadata to allow users to enter a publisher rather than showing a UUID
Added versioning to Workflows
Users with edit permissions on a Workflow will now be able to see an edit button on the Workflow details page which will allow the user to edit the metadata and steps of the Workflow
To support Workflow versioning, users now need to set a version message for Workflow versions
Version tags have been added to Workflow versions, three tags are available - Latest which is automatically assigned, Default and Deprecated. Users with edit permissions will be able to assign a single Workflow version as the default version of their Workflow
Workflow Catalogue prioritises displaying Default versions of a Workflow, it will fall back to the Latest version of the Workflow when no Default is specified.
Added version bar to Workflow version details page to allow users to browse the version history of a Workflow as well as convenient buttons to take you straight to the Default or Latest versions of a Workflow
Added Parameter Sets functionality - this is a large overhaul of the way the Workflow system works and is the reason why we have made this version of DAFNI a new major version
Model Steps in Workflows no longer accept parameter values or datasets, these are now provided in Parameter Sets
When creating a Workflow for the first time you will now need to specify: Workflow metadata, the steps of your Workflow, metadata for the first Parameter Set of your Workflow and the values/datasets for that Parameter Set
Alternatively, when specifying the metadata for a Parameter Set you can select to create a Parameter Set with default values. However, if your Workflow contains uses a Model that has a required parameter that doesn't have a default value provided you won't be able to create a default Parameter Set
The Workflow details page now shows a list of Parameter Sets that have been created for the Workflow, in this section there is a button to allow you to create a new Parameter Set
Next to each Parameter Set in the list are two buttons: an edit button (editing a Parameter Set will overwrite it) and an execute button
To execute a Workflow, a Workflow version and a Parameter Set are now required. The Workflow Version must be saved in the DAFNI catalogue but the Parameter Set does not have to be saved. This is to allow Workflow Instances to be linked to a Workflow version
Workflow Instances are now displayed on the Workflow details page, clicking on the Instance in the list will take you to the status page for the Workflow Instance
Unfortunately, due to the way Workflow Instances used to be stored, we have been unable to match up Instances that were run before this update with Workflow versions. They are still visible on a new page, that can be navigated to through the Workflows catalogue. In a future update we will add functionality to allow you to link historic Workflow Instances up to Workflow versions
New documentation pages have been added to explain how to create a Workflow and Parameter Set in this new system
A YouTube video has also been released to explain the changes that have happened in this release and goes through how to use Workflows and Parameter Sets
Added an info box explaining how globbing of files works for publisher and visualisation steps of Workflows
Improvements
Updated Workflow and Model definitions to use the same casing standard for all fields (snake case)
You can now search for a Dataset or Version ID in the Data catalogue or when searching for a Dataset to use in a model step of a Workflow
Reworked the dialog that appears when executing a Workflow so that there is no delay before it opens to reduce confusion around whether the Workflow has been executed and to prevent repeat executions
The DAFNI web app will now make sure that the step names within a Workflow are unique as if a Workflow contains more than one step with the same name it will fail to execute
Removed tags and type from Model and Workflow metadata while we re-think how to best use these fields in the future
4th August 2021
New
Added "required" flag to Model parameters to allow Model owners to define parameters as optional, this will allow these parameters to be ignored when creating a Workflow
Note: You will still be able to define default values for optional parameters, the NIMS will auto-fill the parameter with the default value if an optional parameter has one defined. An optional parameter will only be fully ignored if no default has been defined and no value is provided in the Workflow
Added role information into pop up that appears when adding a user to a group
Improvements
Moved storage of Model and Workflow definitions from Minio to Postgres - Workflow/Model related pages load faster
Updated all "desc" fields in Model and Workflow definitions to be "description" to keep field names consistent across definitions
Fixes
Removed the misleading pop up that showed after deleting a Dataset
Fixed an issue that stopped the delete buttons on Datasets in a Model Step's Dataslots from working
Removed the error that would briefly show when opening the new plotter tool
Fixed an error with file links that could happen after the plotter was left open for a while
Fixed an issue that prevented users from seeing the full list of parameters and dataslots in a Model step when looking at the status of the Workflow Instance
Fixed an issue that caused users to have reduced permissions when an asset they owned was shared in a group
23rd June 2021
New
The visualisation functionality has been entirely replaced with a new service, the NIVS (National Infrastructure Visualisation Service)
The NIVS has a better back-end which supports future customisation of visualisation options and the addition of new visualisation types
You can now choose a "visualisation type" when creating a visualisation option, instead of being provided with only Jupyter notebooks
A new visualisation type has been added, "Drag and Drop", which uses a simple, browser-based tool for creating 2d visualisations from data, developed in partnership with Martin Robins from Oxford University
A new "visualisations" menu has been added to the frontend which shows a list of all the visualisations which you are allowed to interact with
The "Visualise" button on a finished workflow's visualisation step now redirects to the Visualisations page rather than directly to the visualisation itself. This is not a permanent change and we plan to revert back to linking directly to the specific visualisation in a future release.
Improvements
The version of Jupyter Notebook available for visualisations has been updated to a much more recent version
The model upload process now uses a new backend powered by Skopeo which should be more robust, and will make it easier for us to diagnose failing model uploads
When creating a workflow, the version of a selected model is now displayed to reduce ambiguity
Fixes
Users can now delete a dataset which was created from a workflow publish step (previously, users could only delete versions of a dataset created in this way). Unfortunately, datasets published from workflows from before this change still cannot be deleted by anyone other than a DAFNI admin.
Allows users to change their name and email address on DAFNI
Allows users to see their active DAFNI sessions and log out from all of these sessions from one page
Enables users to add Two-Factor Authentication (2FA) to their DAFNI account
Enables the DAFNI team to explore future benefits like Single-Sign on and Social Login
Login Token will automatically refresh keeping you logged in whilst you're still active on the site
Improvements
Improved security of algorithm used to encode our security tokens
Fixes
Fixed a bug with the cookie which was causing logouts after an hour regardless of the expiry on the login token itself
Fixed a bug where users selected for deletion within a group were still selected even after navigating away from the page and back again
11th May 2021
Improvements
Icon only buttons now have screen reader text to help aid our users in describing what action will take place
Single text action buttons, for example "Add", have been updated to add more meaning: Add data. Editing a Model now reads "Edit model"
Aria-label attributes have been added to form elements. For example, on the Data Catalogue search element, screen reader software will now read out: Search Data Catalogue. The label for sorting the catalogue results now reads out "Sort by". Pagination buttons now read out "Previous page" and "Next page"
The roles information link in Groups is now a button to allow aria attributes to be automatically added
The download icon within a Dataset now reads out the filename when tabbing to the button to help a user understand what could be downloaded
The latest release number in the main navigation now has "Release" added to give meaning to the link
28th April 2021
Fixes
Fixed an issue that caused the Latest version tag to be assigned to the wrong Model version when the Latest Model version was deleted
Fixed a hidden issue where it was possible to add multiple versions of one Dataset to a Dataslot in a Model Step causing undefined behaviour when the Workflow was run. Now if a Dataset version is added to a Dataslot that already contains a different version of the same Dataset, the old version is removed and replaced with the newly selected Dataset version
30th March 2021
New
Links to DAFNI docs, user feedback and social media platforms have been added to the help section
Added optional Source Code field to Model Metadata to allow users to specify a link to the source code of their Model, this link will be displayed on the Model details page
Dataset version message is now displayed on Dataset Details page
Added information to the DAFNI docs website on how different Model parameter types are handled when used in Workflows
Improvements
Dataset and Model Group access permissions flags are now visually different to aid readability
The homepage layout has been updated to improve readability of releases and to bring the look and feel inline with other pages
Group admins alone have permission to remove assets from a group
Assets are now ordered by creation date when you select to remove multiple from a Group
Dataset version information is now displayed in the dataslot table in a Model step
Fixes
Ensure Group error messages are displayed correctly
When editing a dataset the geospatial location, if applicable, is now shown on the page
Correct version message and date are now showing in Dataset version history dialog
Ordering of Model versions in version history dialogs is now consistent
Updated the link injected into Datasets published by Workflows to have the correct URL format
Added a redirect to the Workflow Instance page so old URLs injected into Datasets still work
Empty optional dataslots being rejected by NIMS
16th March 2021
New
Major version update to 2.0.0. See https://dafni.ac.uk/release-2/ for details
Improvements
Model Detail Page now includes Parent model ID and model version message
Updated Delete Model Version dialogue to improve clarity of the action
Group user permissions dialogue updated to reflect current permissions available
Fixes
Users cannot add the same Dataset to a group twice - once added, the Dataset is removed from the list for addition
The number of Models available to be run inside an iterator step is clearly limited to one. This shortcuts a known bug and is a temporary fix until loops are supported in workflows
Group admins who delete themselves from a Group are returned to the main Groups listing page
Changes to the NIMS to ensure smooth operation with back-end metadata validation
2nd March 2021
New
Added the ability for users to be able to delete Model versions and Workflows where they are the owner or have been given administrator privileges to do so
Added an automatic shift of the latest tag of a Model should the latest version be deleted
Added an automatic deletion of the parent Model should the last remaining version of a Model be deleted
Added an error message to alert users when a Model version used in an existing Workflow is deleted, thus rendering that Workflow un-executable
Added a Models tab to the group detail page so that users can share Models with a group as well as see the Models that have been shared with that group
Added a Models tab to the public page so that users can make their Models publicly available
Improvements
Added a user search within groups so that users can be searched for by first name / last name / username rather than requiring a user's ID to add them to the group
Added a link back to the Model version used to run a Workflow in the Workflow status view
When adding a Model or Dataset to a group, users now see text explaining that they are adding the latest available version
The NID API will now run full metadata validation on the backend and return any validation errors in the response
Fixes
Users without sufficient permission to see the latest version of a Model will no longer see the "Go to latest" button
The "Don't forget to set a version" error message is now properly cleared if the problem is addressed
Members of a group who are non-admins can no longer see the edit button in the group details page
All members of a group can now add and remove assets from the group
Can no longer open multiple Dataset selection dialogs simultaneously when modifying a Model step in a Workflow
Fixed an issue that prevented a Model version that failed to ingest from having a new version uploaded
Optional dataslots are no longer required to be filled with a Dataset in Model steps
Optional dataslots with no default supplied now load correctly on the Model details page
2nd February 2021
New
Workflow metadata page introduced which provides a cover page to Workflow information
Improvements
Combined the upload and publish actions on the Add Data page into one, meaning users no longer need to wait for their files to finish uploading before pressing publish
Added a progress bar to the Data uploading dialog
Added a cancel button to the Data uploading dialog that will allow users to stop the creation of a Dataset if they wish to go back and change metadata or files
Changed the behaviour of the Dataset metadata form, the publish button will no longer be disabled, instead users will be able to press it even with invalid metadata, the form will be validated at that point and any errors highlighted
20th January 2021
New
Group details (name and description) can now be edited by group administrators
Added versioning functionality for Models
Model Owners can now see an edit button on the Model details page that will allow them to upload a new Model Version
When uploading a new Model Version for an existing Model you can provide a Model Definition, Model Image or both
To support Model Versioning users are now required to add a version message when uploading a Model
Added version flags to Model Versions, three flags are available - Latest (automatically assigned), Default and Deprecated. Model Owners are able to assign a single version to be the Default version of their Model and any number of versions to be Deprecated
The Model Catalogue will prioritise displaying the Default version of a Model, falling back to the Latest version of a Model when there is no Default specified. Currently this means that these versions will be the only ones available to users in Worfklows, a future update will be adding functionality to select which version of a Model to use in your Workflow
Added a version bar to Model details page regular users browsing Models will be able to use the bar to browse the versions of Models they can see, via a version history dialog and 2 convenience buttons to take you straight to the Default or Latest versions of the Model
Model Owners will see extra buttons in this bar that will allow them to assign/remove the Default and Deprecated flags to/from versions
Improvements
DAFNI blue colours have been darkened slightly to better meet AA website accessibility compliance
Increased JWT expiry time to be 4 hours in order to reduce the likelihood of users being logged out mid-upload
YAML and JSON problems in the Model definition file are now much more clearly conveyed to the user with descriptive error messages
Error is much clearer when Model definition references a Dataset that doesn't exist in DAFNI
Added tooltips to Model parameters within a Workflow
Fixes
Dataset version upload will now not take files from the previous version if new files are uploaded
Fixed an issue where uploading two Datasets one after the other would fail due to a cached bucket ID
Fixed an issue where uploading a valid YAML file during Model upload after first uploading an invalid one would still result in an error due to the error being cached
12th January 2021
Improvements
All users are now visible in a group and the owner clearly identified
Information on Group roles is now available with details on what permissions can be applied to a user in a group
Plotly (a Python package for drawing plots) is now available in Jupyter Notebook visualisations
Model step parameters with minimum and maximum values set will be validated upon step creation
Fixes
Fixed an issue where upon selecting a Dataset in groups and paginating the selection would not work correctly
24th November 2020
Improvements
Updated indexing strategy to pre-calculate Dataset file formats during Dataset ingestion to improve data catalogue search speed
Updated filters lists to be ordered by the filter value's count
Fixes
Removed zero counts from filter lists
Fixed an issue that caused Dataset version update notes to not be visible in Dataset version dialog
Fixed an issue that prevented navigation to a previous version when using Dataset version dialog
10th November 2020
New
Manage permissions on Datasets within a group
Manage users within a group and set appropriate roles to each user
Allow users the option to leave a group
Added Groups to allow the creation and management of a group of users to enable collaboration on private Datasets
Allow Group administrators the ability to:
Manage Datasets within a group
Changed Model definition Datasets section to Dataslots, this allows Model owners to specify a list of default Datasets to use for each slot in the Model. Each slot loads Datasets to a path specified by the Model owner when the Model is used in a Workflow
The move to Dataslots allows users of the Models to change the Datasets a Model runs with when they create a Workflow
Added section in Model steps to allow selection of Datasets to use with a Model via searching the Data catalogue
Added a check on Model upload to confirm that Datasets specified in the Model definition exist
Improvements
Updated step validation in Workflows to trigger when clicking Apply Changes or Create Step, making it more obvious what mandatory fields are not filled in.
Minor UI improvements in the Workflows builder
When deleting an entry from editable tables a DAFNI confirmation dialog is now shown, rather than a browser based one
Fixes
Fixed bug where step name was showing instead of Model name in the summary of a Model step when viewing a Workflow instance
Removed old example notebooks and data from Jupyter Notebook Visualisations
Fixed the inconsistency between pages when displaying Model parameters. All pages now show human readable Model parameter titles
Fixed a bug where selecting the "Use metadata file" option in a Publisher or Visualisation step would cause the step to not appear in the Workflow instance view
Fixed a bug where re-selecting the same Model when in a Model step would remove all parameters from the step
27th October 2020
Improvements
Add hint text to Workflows page explaining how to pan the Workflow canvas
Add pagination to the Dataset version dialog
Added buttons to allow users to copy Dataset and Version UUIDs from the Dataset details page
Provide information on reason for Dataset access permissions on Dataset details page
Fixes
Fixed a bug which was causing empty files to be written into Jupyter Notebooks
Re-added tooltips for Model parameters to the Model details page
Fixed the way the geojson is created for points in the Dataset spatial coverage component
Fixed Mac users being unable to add .tar.gz files when uploading a Model
Fixed a bug that caused "Apply changes" button to remain disabled when editing Publisher and Visualisation steps in Workflows
Overhaul of Dataset details and Dataset upload pages to support new metadata schema, improve the look and feel and provide a better user experience
Added versioning capability to Data Catalogue to allow a Dataset to have multiple versions that can be explored through the Dataset details page
Added an edit button to Dataset details page that will allow admins of a Dataset to add a new version, either with new Data files or just to change some of the metadata
Overhaul of Data Catalogue to support Elasticsearch to provide comprehensive new search functionality, improve the look and feel and provide a better user experience
Added filtering options - temporal, source, subject and file type filters with counts for how many Datasets you can see that match the filter
Added sorting of Data Catalogue by recent, relevance to search criteria and alphabetical
Updated other DAFNI components (NIMS, NIVS, NICE, DSS, etc) to support the NID overhaul
Improvements
Optimisation of NID calls to improve their speed
Fixes
Fixed the time in the Workflow execution pop up so that it is in local time rather than UTC
1st June 2020
New
Added the list of Datasets a Model uses to its metadata page
Added more detail to the Model upload documentation
Added validation of values provided to Model parameters in Model steps when a Workflow is created or executed
Added an option to use a GPU with a Model to the Model definition
Added Model version flags - 'default', 'deprecated' and 'latest' to public Models
Added a Release Notes page to the user documentation site
Improvements
Optimisations of DSS and NIMS calls to improve their speed
Fixes
Fixed a bug where, if multiple steps in a Workflow tried to use the same Dataset, the Workflow would not be executed correctly
1st May 2020
New
Added a progress bar to Model upload pop up
Fixes
Fixed a bug where Model parameters that didn't specify options were not being shown in the parameter configuration table for Workflow Model steps
Added a loading screen after pressing 'Create Workflow' to prevent users accidentally creating multiple Workflows
Fixed the cancel button on the Model upload pop up so that it actually cancels the upload
Fixed a bug where switching from taking metadata from a file to specifying it yourself in Workflow Visualisation and Publish steps would cause the step to be treated as valid resulting in the creation of invalid Workflows
1st April 2020
New
Added functionality to allow a user to upload their own Model to DAFNI, currently uploaded Models will only be visible to the user that uploads them
Added a user documentation site to provide in depth documentation for some parts of DAFNI, such as the Model upload process
Added a repository of example Models that are ready to be uploaded to DAFNI
Fixes
Fixed a bug where a leading '/' in the Model definition Dataset path would cause Model runs to fail
27th February 2020
New
Added status flag to Models in the NIMS catalogue. This shows whether a Model has been uploaded to the catalogue correctly and is available to be used in Workflows.
Added panels for release notes and Twitter feed to DAFNI's welcome screen
Added drag and drop area for file selection to the NID's add Dataset page
Added tooltips to the NID's add Dataset page, these are also visible when adding a publisher or visualisation step to a Workflow
Improvements
Major Upgrade of the component framework we use that has resulted in some slight visual changes
Fixes
Fixed the layout of the dialogs in DAFNI so that they are all aligned
Fixed Workflow publishing step so that it handles structured outputs from Models correctly
6th December 2019
New
Added Manage Access tab that will allow users to request a change of visibility of the Dataset they upload
Fixes
Fixed the layout of Dataset metadata to align with how Model metadata appears
Fixed a bug where there was no loading icon on Model metadata
Fixed a bug that meant the Dataset list would take a long time to load
Various minor UI changes
31st October 2019
New
Switch to filter out Dataset that was published by Workflows from the NID catalogue
Unified terms within the facility, we noticed that the way we were referring to things wasn't consistent across the interface so have made an effort to improve that consistency
Added list of input parameters to the Model metadata page
Added list of output files to the Model metadata page
Streamlined the loading of Workflow instances to reduce loading time
Made use of the Model catalogue and metadata pages when choosing a Model in a Workflow's Model step. Input parameters are now automatically drawn from the Model's metadata to assist in the creation of Workflows
Loading icons to Workflows and Workflow instances catalogues
Optimised loading metadata catalogue substantially reducing loading times
Fixes
Fixed a bug where uploading Dataset that had symbols in the file names could result in the upload failing
Fixed a bug that meant Dataset in the NID catalogue couldn't be ordered by name
Fixed a bug where the loading of Dataset files on the metadata page would prevent the Dataset's metadata from being displayed
10th September 2019
New
Account details page that lists First, Last Name, Email Address and UUID for DAFNI
Account details allows for updating of users password with validation of the password for STFC password rules
Login page allows users to say they have forgotten their password which will then email a reset your password link
Newly uploaded Dataset are now only visible to the user that uploaded them
Dataset can be marked as View only in the NID, metadata page will display a message telling a user that they must request access to the Dataset and will not allow the user to download the associated Dataset files
Workflows template steps have been simplified, their full functionality is not implemented yet so their use has been temporarily restricted to within iterators
6th August 2019
New
NID, National Infrastructure Database catalogue, a searchable, ordered table of Dataset with customisable pagination
Dataset Metadata pages, every Dataset in the NID has metadata associated with it that can be viewed by clicking on it in the NID table
Download Dataset, the metadata page also displays the list of Dataset files that are a part of the Dataset clicking on these lets you download the file if you have appropriate permissions
Add Dataset, can upload Dataset, with metadata attached to the NID, any Dataset uploaded at this stage will be owned by and only visible to the user that uploaded it
NIMS, National Infrastructure Modelling Service, 2 searchable, orderable tables with customisable pagination one for Models/transformations and one for Workflows
Model Metadata pages, every Model in the NIMS has metadata associated with it that can be viewed by clicking on it in the Models table
Workflow Instances, Workflows in the NIMS can be executed, by clicking on the execute button in the Workflows table, if you have appropriate permissions on execution an entry will be added to the Workflow Instances table
Editing Workflow, clicking on a Workflow will take you to an editable visual representation of the Workflow
Adding steps, steps can be added to Workflows by clicking the plus icon
Editing steps, the Dataset in the steps of a Workflow can be edited from the visual representation
Saving Workflows, Workflows can be saved to the NIMS - this creates a new entry in the Workflows catalogue
Executing Workflows, Workflows can also be executed from the Workflow's visual representation this will add a new entry to the Workflow Instances table
Creating Workflows, brand new Workflows can be created by clicking the Create Workflow button above the Workflows table
Wokflow Instance status, a quick summary of the status of a Workflow can be seen by clicking on the entry in the table, the row will expand displaying each step and its status broken down by the number of processes within it
Workflow Instance status extended, a visual representation of the status of a Workflow can be seen by clicking on the title of the Workflow instance, this will display a static version of the visual representation from editing Workflows. Icons will be visible displaying the status of each step in the Workflow, the summary of the step and its status by process can be seen by clicking on the step
Results Visualisation, if a Workflow has a visualisation step then the visualisation can be accessed through the Workflow Instance visual representation, by opening the visualisation step summary and clicking the "Visualise..." button