I’ve been exporting a lot of data from my ServiceNow instance lately for archival purposes (IYKYK) and one of the challenges I’ve encountered is properly preserving attachments.

Converting from Base64?

At first I looked for ways to “re-hydrate” the attachments from records exported in XML format. Attachment files are embedded in XML exports in a kind of Base64 encoding that’s probably not proprietary to ServiceNow, but I tried several different ways to convert that encoded data back into files with no luck. If you know a way to do this, please reach out and share, as I’m still curious to know if it can be done.

Server-side script and/or Processor?

The next approach I took was writing a server-side script (like a Fix Script or a Background Script) to initiate bulk downloading of attachments in my browser. I stumbled across a few promising Community posts going in a similar direction, including one that suggested creating a UI Action that utilized a custom Processor. That and similar answers all seemed to be linking to a now-missing post on the ServiceNow Guru blog.

There’s apparently an out-of-box Processor called DownloadAllAttachmentsProcessor that will (what it says on the tin) download all attachments for a single record as a zip file. Handy! But I needed something that would download attachments in bulk, and custom Processors are not only deprecated but completely disabled (you need the maint role to create them and you can’t edit the ACLs) so all this was a dead end.

Attachment API and Python to the rescue

Fortunately, all that thinking about Processors got me on the related subject of REST APIs. At first I thought I might have to create a Scripted REST API, but then I discovered there is an out-of-box Attachment API. The last missing piece fell into place when I found Hitoshi Ozawa using a Python script to hit the Attachment API from the command line on his local machine.

Ozawa’s solution seemed promising, but he was still only downloading the attachments on a single record. So here’s my own version of a Python script that downloads all attachments for all records in a specified table matching a specified encoded query, organizing them in a nice folder hierarchy (don’t miss some notes on usage after the script, including a very important one about security):

import requests
import os
import re

### set options here ###

instance = 'https://<your-instance>.service-now.com'
username = ''
password = ''

table = 'incident'
display_field = 'number'
query = 'active=true'
limit = '5' # set to 'None' (without quotes) to remove limit

directory = './Attachments'

### shouldn't have to edit anything below ###

# helper dict for converting mime types to file extensions
mime_type_to_ext = {
  'application/json': ['.json'],
  'application/xml': ['.xml'],
  'image/png': ['.png'],
  'image/jpeg': ['.jpg', '.jpeg'],
  'image/gif': ['.gif'],
  'text/csv': ['.csv'],
  'text/plain': ['.txt'],
  'application/pdf': ['.pdf'],
  'application/msword': ['.doc'],
  'application/vnd.openxmlformats-officedocument.wordprocessingml.document': ['.docx'],
  'application/vnd.ms-excel': ['.xls'],
  'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet': ['.xlsx'],
  'application/application/vnd.ms-powerpoint': ['.ppt'],
  'application/vnd.openxmlformats-officedocument.presentationml.presentation': ['.pptx'],
}

# helper function for hitting any endpoint, returns response object
def get_response(endpoint, mime_type, params={}):
  headers = { "Accept": mime_type }
  response = requests.get(endpoint, auth=(username, password), headers=headers, params=params)
  if response.status_code != 200:
    print('Status:', response.status_code, 'headers:', response.headers, 'Error Response:', response.json())
    exit()
  return response

# helper function for table api, returns list of dicts
def get_records(table, query, limit=None):
  endpoint = f'{instance}/api/now/table/{table}'
  params = {
    'sysparm_query': query,
    'sysparm_display_value': 'true',
  }
  if limit:
    params['sysparm_limit'] = limit

  return get_response(endpoint, 'application/json', params).json().get('result')

# helper function to make directory and file names safer
def sanitize_filename(filename):
  filename = re.sub(r'[<>:"/\\|?*]', '', filename) # remove invalid characters
  filename = re.sub(r'^\.+', '', filename) # remove leading period
  return filename[:255] # truncate to 255 characters
  
# Helper function to generate a unique filename if file exists
def unique_filename(path):
    base, extension = os.path.splitext(path)
    counter = 1
    while os.path.exists(path):
        path = f"{base}_{counter}{extension}"
        counter += 1
    return path

# helper function for attachment api, writes files to disk
def get_attachment(sys_id, mime_type, path):
  path = unique_filename(path)
  endpoint = f'{instance}/api/now/attachment/{sys_id}/file'
  response = get_response(endpoint, mime_type)

  with open(path, 'wb') as f:
    for chunk in response.iter_content(chunk_size=1024):
      if chunk:
        f.write(chunk)

# call table api to get specified table records and iterate over results
for record in get_records(table, query, limit):
  class_name = sanitize_filename(record.get('sys_class_name'))
  display_value = sanitize_filename(record.get(display_field))
  print(f'{class_name}: {display_value}')

  # call table api again to get attachment metadata and iterate over results
  attachment_query = f"table_sys_id={record.get('sys_id')}"
  attachments = get_records('sys_attachment', attachment_query)

  if attachments:
    # make folders if they don't exist
    path = os.path.join(directory, class_name, display_value)
    os.makedirs(path, exist_ok=True)

    for attachment in attachments:
      # get mime type and file name
      mime_type = attachment.get('content_type')
      file_path = os.path.join(path, sanitize_filename(attachment.get('file_name')))

      # Append file extension to file name if not already
      file_ext_list = mime_type_to_ext.get(mime_type, [''])
      file_ext = file_ext_list[0]  # Choose first extension as default
      if not any(file_path.endswith(ext) for ext in file_ext_list):
        file_path += file_ext

      # call attachment api to download file and write to disk
      get_attachment(attachment.get('sys_id'), mime_type, file_path)

Usage notes

Here’s how to use the script:

  1. Copy and paste it into your favorite text editor (worth noting here: unlike in JavaScript, indentation matters in Python, so copy carefully)
  2. Update the settings at the top of the script to your liking
  3. Save it as attachments.py (or whatever you want)
  4. On the command line, navigate to the same folder where you saved the script
  5. Run it with the command python attachments.py (you might have to start the command with python3 depending on your environment)

When you run the script, it creates an “Attachments” folder in the same location. It then begins downloading the attachments into that folder, neatly organizing them in a folder hierarchy by table name and record display value. It also helpfully adds appropriate file extensions to any files that don’t have them.

I can’t put a fine enough point on this: It’s super dangerous to put your ServiceNow username and password directly in a plain text file like this. If you’re using it as a one-off like I am, it may be fine, though I’d be sure to delete the password from the script (or just delete the whole script) when you’re done. If this script is going to be used on a regular basis you should investigate dynamically populating the username and password from environment variables or an encrypted password vault. Also, if you can get what you need from a sub-production instance, it’s always safer to go that route. If my script gets you reprimanded or fired over some kind of security breach, don’t @ me.

Lastly, you’ll (obviously) need Python installed on your local machine to run this. Instructions for doing that are outside the scope of this article, partly since they’ll be different depending on your operating system. In addition to Python, I found I had to separately install the requests package using the package installer for Python (pip). Google is your friend for figuring out how to get everything installed and running on your computer.

Questions or feedback?

If you find a use case for downloading attachments in bulk, I hope the script helps. I’d love to hear about your experience and answer any questions you might have. Feel free to reach out on SNDevs Slack or Mastodon. Cheers!