Update (23/02/2023) - 'Webhook on item expiration' - Release postponed
Since first announcing the 'Webhook on item expiration' capability at Microsoft Build 2022 (see c.17 minutes into the following video: Automate and customize retention and deletion scenarios - Events | Microsoft Learn) Microsoft has unfortunately decided to postpone the creation of this feature. It is not currently known when this really exciting capability will now be available.If you’ve got anything more than the lowest level compliance needs, planning retention in Microsoft 365 can be a real challenge. Not only do you need to translate all of your retention periods into retention labels and/or policies and establish how they will be applied to content, you also need to determine what happens at the end of retention.
For the past 5 years the options that we’ve had at the end of retention have been fairly unchanged – but over the past couple of months Microsoft has unveiled several new options. Given the amount of change in this area, I thought it would be useful to put together this blog post, which details both the existing and new approaches that we can take advantage of in our Microsoft 365 architectures.
The limits of retention policies
Cards on the table, I’m not personally an advocate of retention policies. I know many like to use them for their simplicity, but I feel that the risks associated with accidently having content retained behind the scenes which your staff think they’ve deleted largely make retention policies incompatible with GDPR and in my view largely preclude their use. My only exception to this is for workloads such as Yammer and Microsoft Teams Chat, where they are the only option that you currently have when it comes to retention.
Retention policies are fairly limited when it comes to what you can do at the end of retention, only presenting you with two options:
Do nothing
The most basic option for the end of retention, which means that nothing, well almost nothing, happens at the end of the retention period.
Microsoft recently renamed this option for retention labels to ‘Deactivate retention settings’ (but in my tenant at least it still retains its former name of ‘Do nothing’ for retention policies). This new name is technically more accurate, as while at the end of retention nothing happens to the content itself, the rules determined via the retention label/policy will stop applying. As an example, this means that the retention rules that might have prevented a file from being deleted during its retention period stop applying at the end of retention, after which time the file can be deleted by users without being retained.
Personally, I rarely use this option – it doesn’t help us fulfil any of the most frequent records management needs. For example it doesn’t allow us to review, delete, or archive our content. I’m sure there will be some niche scenarios where it’s really useful (do let me know if you’ve stumbled across any!).
Delete items automatically
At the end of the retention period another process starts that will delete content automatically. At Intelogy, we spent a little bit of time last year testing how long this process took to run (specifically for Microsoft Teams Chat) and while content was typically deleted in under a week after the end of retention, Microsoft informed us that it can take up to 16 days in some scenarios (see my blog on: How do you solve a problem like Microsoft Teams Chat and Posts retention?).
This is obviously a great option to have – it’s perfect for any content we want to make sure is deleted on a timely basis. In most organisations c. 60-80% of content will likely fall into this category and need to be subject to an automated deletion process.
These are the only two options you have with retention policies. There is no option to review or consider items that need to be transferred to archive – you can merely automatically delete or chose to do nothing. The good news is that retention labels provide far more flexibility!
These are the only two options you have with retention policies. There is no option to review or consider items that need to be transferred to archive – you can merely automatically delete or chose to do nothing. The good news is that retention labels provide far more flexibility!
What options do I have with retention labels?
The first thing to mention is that retention labels also provide both of the same end of retention options as retention policies – the only difference is that they provide a wealth of other alternative approaches too. In addition to being able to ‘Do nothing’ or ‘Delete items automatically’, retention labels also let us choose from the following options at the end of retention:
Start a disposition review
At the centre of most of my retention architectures in Microsoft 365, this approach triggers a process that allows content to be reviewed at the end of retention. The idea is that you have an opportunity to appraise some of your more important content to determine its ongoing value and whether it is sent to archive or deleted.
Different retention labels can be associated with different groups of reviewers – something that I’ve used to ensure that content is only sent to the relevant users for assessment. This is invaluable in scenarios where multiple organisations share the same tenant, ensuring that content is reviewed by staff working in the relevant organisation. Before this ability to delegate disposition review was introduced last year it presented a significant gap in the retention story in Microsoft 365.
You can have up to five sequential stages of review, with different reviewers at each stage. However, the disposition review process is very focused on ‘deletion’, with the only way of moving from one stage to another being to mark the item for disposition. I really wish that Microsoft would extend this process further through providing the ability to mark files for transfer to an archive – something that I feel would prove invaluable for many organisations.
One of the major downsides of triggering a disposition review is that it requires E5* licences for all users who can modify content that is subject to the a review label – in many (but not all) scenarios this means that all users need a premium licence.
* or Microsoft 365 E5 Compliance/ Microsoft 365 E5/A5 Info Protection & Governance
Change the label
Only introduced a few weeks ago, this capability allows you to chain retention labels together, so when the first reaches the end of retention a different label is automatically applied. This is actually more useful that it might at first appear. For example, you might have files tagged with a standard retention label, which staff can modify for the first few of their life. When this label reaches the end of its retention period, you could chose to automatically re-classify the content with a record label, making those same files immutable for a further 5 or 10 years.
Any new retention option is more than welcome - and generally I feel that there will be some scenarios where changing the label is a very useful approach. I do wish that we could conditionally determine which label is applied though – perhaps allowing us to automatically apply one label or another based upon the content’s other metadata – as this would introduce significantly more flexibility.
Run a Power Automate flow
Another option introduced in May this year that I can see is certainly going to see extensive use. This approach allows us to automatically trigger a Power Automate process at the end of an item’s retention period. For those of you who aren’t familiar with Power Automate – it’s Microsoft 365’s workflow solution, which makes it incredibly easy to implement your own dynamic business processes.
Still in preview, Microsoft has introduced a new Power Automate trigger, which will run whenever an item in SharePoint reaches the end of its retention period. Supplementing this are two new workflow actions – the ability to delete items at the end of retention and the ability to move content between SharePoint sites.
At the moment this workflow solution seems to only function for content in SharePoint – I’d need to test to see what happens when content in other workloads reach the end of retention (my guess is nothing at the moment – however, I’d be surprised if Microsoft didn’t quickly extend this capability to function wherever a retention label has been applied).
This approach is obviously the most flexible we have to date – effectively we can implement any business process we want at the end of retention. We could automatically send notifications, or tag content, or check metadata, or move files from one location to another.
Now, I’ve not seen it documented, but I’ve been told that this capability requires Power Automate premium licences (if anyone from Microsoft is reading, please let me know!). The problem is, I don’t know how many licences will be needed. If I understand it correctly, it will depend entirely on the ‘value’ that is derived from the process:
“If the flow uses premium connectors and only the owner is getting the value from the flow, since the trigger is an automated trigger, only the owner needs a premium license.”
Source: Frequently asked questions about Power Automate licensing - Power Platform | Microsoft Docs (my emphasis)But who is getting value from running a process at the end of retention? Is it the records management team who might otherwise need to run a manual process? Or is it everyone in the organisation? The answer is critical – as at £11.30 per user/month the cost could quickly add up.
One alternative might be to consider a per flow plan which is currently priced at £4,524 per year for 5 premium flows – which is very likely worth considering.
Webhook on item expiration
Update (23/02/2023) - Release postponed
Since first announcing this new approach to ending retention at Microsoft Build 2022 (see c.17 minutes into the following video: Automate and customize retention and deletion scenarios - Events | Microsoft Learn), a decision has been taken to postpone the creation of the 'Webhook on item expiration'. It is not currently known when this really exciting capability will now be available.I’ll be honest, I was absolutely delighted to see Microsoft provide us with a webhook. This is something I’ve been asking Erica Toelle and Roberto Yglesias to provide in addition to the Power Automate connectors over the past year – and I’m excited by the flexibility and extensibility offered by this approach.
For those of you who aren’t developers, a webhook is effectively a custom trigger, that allows us to attach our own logic and code to it. This means that when an item reaches the end of retention, the webhook will fire and allow us to call into a custom API and invoke any process we’d like to. Without a doubt, this will be my preferred approach for automating the most complex post-retention activities, such as undertaking security reviews and managing transfer to archives.
It's important to note that this option will not (initially at least) be show in the UI – if we want to make use of webhooks, we’ll need to set them up with PowerShell:
The end of the beginning?
It’s always great to see existing tools being made more flexible. These new options for the end of retention represent a substantial improvement, that will enable organisations to implement their own processes at the end of retention. These changes remove many of the constraints around the retention process in Microsoft 365, by providing us with the ability to adapt the platform to meet our own specific retention needs. To help summarise all of the end of retention options, I thought I'd put them into an infographic:
I’d be interested in hearing your thoughts about these new capabilities. What new options for the end of retention would you like to see Microsoft introduce?
As ever, I’m always happy to help answer your retention and compliance questions – why not get in touch?
This article was originally published on Intelogy - Microsoft Gold Partner, Office 365, SharePoint Experts
Comments