Automating GIS Workflows with Python Scripts

car assembly line

This week I am presenting a session titled Automating Workflows with Python at the Pennsylvania GIS Conference.  The session walks people through the process of creating a Python script that can be set-up as a Windows Scheduled Task.  This would allow simple tasks like data replication and rebuilding address locators to automatically run on a weekly basis.  The script features writing the results and/or errors of the geoprocessing tasks to a text file.  I wanted to share this presentation in a blog format.  I’m also going to share the links to the presentation and the re-usable Python template script:

One thing I mention in my presentation is that this process can be used by both developers and non-developers. For those weary of Python, you can create a model in ModelBuilder and export the model to a Python snippet. You could then take the Python snippet and place it in the template script I’ll be reviewing in this post.

Reviewing the Script

workflow diagram of python script for data replication

The first step is importing various modules.  We use ArcPy to perform ArcGIS geoprocessing, sys and traceback to handle errors, time to time how long the processes take, and datetime to reformat the current date.  As I’ve built this script from various samples, I’m actually not sure os and string are neccassary.

 

# Import system modules
import arcpy
import os, sys, time, datetime, traceback, string

 

The next step is to get the current date so we can append that to the empty text file we’ll be creating to log the results/errors. datetime.now() returns the current date and time (as in 2017-05-11 19:06:58.367814). strftime() allows us to reformat the current date and time.  My script includes an example for the date alone, as well as both the date and time.  The second option is ideal if your script runs multiple times a day.  My scripts typically run weekly, so just the date is fine.  Calling strftime(‘%m-%d-%Y’) would return 05-11-2017.

 

# Time stamp variables
currentTime = datetime.datetime.now()
# Date formatted as month-day-year (1-1-2017)
dateToday = currentTime.strftime("%m-%d-%Y")
# Date formated as month-day-year-hours-minutes-seconds
dateTodayTime = currentTime.strftime("%m-%d-%Y-%H-%M-%S")

 

The next step is creating a new text file and opening it in write mode.  The dateToday variable is included in the file name.  This way, each time the script runs, the date will be included in the name.  If the file will reside on a network drive, I would recommend using the UNC path instead of the letter drive.

UNC paths seems to work better when the script is run as a scheduled task. If you copy the letter drive path into Microsoft Word or Outlook, create a hyperlink to that path, and then click on the link, it will open the folder with the UNC path.

We’ll use the open() method to open the file in write mode.  This method takes parameters for the file, mode (read, write, append, etc), and optional buffering.

 

# Create text file for logging results of script
# Update file path with your parameters
# Each time the script runs, it creates a new text file with the date1 variable as part of the file name
# The example would be GeoprocessingReport_1-1-2017
file = r'C:\GIS\Results\GeoprocessingReport_{}.txt'.format(dateToday)

# Open text file in write mode and log results of script
report = open(file,'w')

 

Now we’re ready to run the geoprocessing. We’ll run this code in a Try/Except block.  As I’m writing this, I wonder if the creation of the file should be done within a Try statement as well.  Any thoughts?  We store the geoprocessing tool in a variable to get access to the status code and message.  After the tool completes running, we’ll write two messages to the log file:

  1. The ArcGIS system message which can be confusing to read
  2. A human friendly summary of the tool’s completion

 

# Run geoprocessing tool.
# If there is an error with the tool, it will break and run the code within the except statement
try:
    # Get the start time of the geoprocessing tool(s)
    starttime = time.clock()

    # SDE is parent geodatabase in replication
    # Change this to your SDE connection
    sde = r"SDE Connection"
    # Child file geodatabase in replication
    # Change this to your file geodatabase
    child_gdb = r"\\path\to\file.gdb"

    # Process: Synchronize Changes
    # Replicates data from parent to child geodatabase
    result = arcpy.SynchronizeChanges_management(sde, "Name of Replication", child_gdb, "FROM_GEODATABASE1_TO_2", "IN_FAVOR_OF_GDB1", "BY_OBJECT", "DO_NOT_RECONCILE")

    # Get the end time of the geoprocessing tool(s)
    finishtime = time.clock()
    # Get the total time to run the geoprocessing tool(s)
    elapsedtime = finishtime - starttime

    # write result messages to log
    # delay writing results until geoprocessing tool gets the completed code
    while result.status < 4:
        time.sleep(0.2)
    # store tool result message in a variable
    resultValue = result.getMessages()
    # write the tool's message to the log file
    report.write ("completed {} \n \n".format(str(resultValue)))
    # Write a more human readable message to log
    report.write("Successfully ran replication from {} to {} in {} seconds on {}".format(sde, child_gdb, str(elapsedtime), dateToday))

 

The final parts of our script are the Except statements and closing the text file.  I include both an Environment Error and Exception error types.  From what I understand, Environment Errors occur outside of the Python system.  I use it as backup error handling.  The Exception block would handle errors related to the geoproccessing tool.  Finally, we close to log file as good practice.

 

# If an error occurs running geoprocessing tool(s) capture error and write message
# handle error outside of Python system
except EnvironmentError, ee:
   tbEE = sys.exc_info()[2]
   # Write the line number the error occured to the log file
   report.write("Failed at Line %i \n" % tbEE.tb_lineno)
   # Write the error message to the log file
   report.write("Error: {}".format(str(ee)))
   # handle exception error

except Exception, e:
   # Store information about the error
   tbE = sys.exc_info()[2]
   # Write the line number the error occured to the log file
   report.write("Failed at Line %i \n" % tbE.tb_lineno)
   # Write the error message to the log file
   report.write("Error: {}".format(e.message))

# close log file
report.close()

 

So that is a summary of the standard script template I use for my GIS scripts that will be set as scheduled tasks.  And to close, here are some things we are automating at Cumberland County:

  • Data replication to various department file geodatabases
  • Rebuilding address locators used in our geocoding service
  • Rebuilding tiles for cached map services
  • Monthly data updates to a regional group of counties (extract data to geodatabase, zip geodatabase, upload to FTP)

If you’re interested in viewing the slides from my presentation, or grabbing a copy of the template script, see these links:

Advertisements

One thought on “Automating GIS Workflows with Python Scripts

  1. You may want to consider using try-except blocks for creating a file. Let’s consider a situation where the file is getting created on a network drive. If the network connection is interrupted, the creation of your file will fail and the program will crash.

    Also, if the script is getting run in a read only directory and trys to create a file in the current directory, your script will fail when it tries to create the file also. So unless you know that the file creation will always be successful, it’s not the worst idea to protect your file creation code.

    Great post!

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s