Stricter output paths in project.yaml
07 November 2023
We’ve made some changes to slightly restrict output file paths.
Specifically, all paths must now end with file extension, e.g. .csv
.
Previously, it was possibly to use a trailing *
character to match all files
(e.g. outputs/data.*
). You can still use *
to match groups of similar
files, but you must end the pattern with a file extension (e.g.
outputs/data*.csv
).
Nearly all project.yaml files do already have explicit file extensions in all their output paths, and this will not require any changes for most users. There are a few users who may need to add an explicit file extensions in order for jobs to run, both locally and on backends. The opensafely tool that you run locally has been updated, and it will warn you that you need to do this.
There are multiple motivations for this change, but the main one is that we
want to be able to inspect the file types when loading a project.yaml. This
allows us to warn users if they are trying to mark an unsupported file
type
as moderately_sensitive
.
It also stops accidentally capturing output files that were not intended to be outputs (which incurrs a performance and disk space cost).
Any questions or problems, please let us know.