Automating GIS Workflows with Python Scripts

car assembly line

This week I am presenting a session titled Automating Workflows with Python at the Pennsylvania GIS Conference.  The session walks people through the process of creating a Python script that can be set-up as a Windows Scheduled Task.  This would allow simple tasks like data replication and rebuilding address locators to automatically run on a weekly basis.  The script features writing the results and/or errors of the geoprocessing tools to a text file.  I wanted to share this presentation in a blog format.  I’m also going to share the links to a recording of the presentation, the slides, and a sample template Python script:

One thing I mention in my presentation is that this process can be used by both developers and non-developers. For those weary of Python, you can create a model in ModelBuilder and export the model to a Python snippet.  You could also go to the Results tab of the Geoprocessing toolbar and copy a completed tool as a Python snippet. You can then take the Python snippet and place it in the template script shared above.

Reviewing the Script

workflow diagram of python script for data replication
Diagram showing the flow of the template Python script

 

The first step is importing various modules.  We use ArcPy to perform ArcGIS geoprocessing, sys to handle errors, time to calculate how long the processes take, and datetime to reformat the current date.

# Import system modules
import arcpy, sys, time, datetime

The next step is to get the current date so we can append that to the empty text file we’ll be creating to log the results/errors to. The datetime.now() method returns the current date and time (as in 2017-05-11 19:06:58.367814). The strftime() method allows us to reformat the current date and time.  My script includes an example for the date alone, as well as both the date and time.  The second option is ideal if your script runs multiple times a day.  Calling strftime(‘%m-%d-%Y’) would return 05-11-2017.

# Time stamp variables
currentTime = datetime.datetime.now()
# Date formatted as month-day-year (1-1-2017)
dateToday = currentTime.strftime("%m-%d-%Y")
# Date formated as month-day-year-hours-minutes-seconds
dateTodayTime = currentTime.strftime("%m-%d-%Y-%H-%M-%S")

The next step is creating the variables logMsg and logFilelogMsg will contain the content we want to write to the text file. logFile will be an empty text file.  At the end of the script, within the finally statement, we will open logFile in write mode, and write the contents of logMsg to the text file.  We will use the += operator to keep appending messages to the logMsg variable.

If your text file, or any other file paths use networked drives, I recommend using the  UNC path instead of the letter drive.  UNC paths seems to work better when the script is run as a scheduled task. If you copy the letter drive path into Microsoft Word or Outlook, create a hyperlink to that path, and then click on the link, it will open the folder with the UNC path.

# Each time the script runs, it creates a new text file with the dateToday variable as part of the file name
# The example would be GeoprocessingReport_1-1-2017
logFile = r'C:\GIS\Results\GeoprocessingReport_{}.txt'.format(dateToday)

# variable to store messages for log file. Messages written in finally statement at end of script
logMsg = ''

# add content to logMsg
logMsg += 'Here is a message for the text file\n'

Now we’re ready to run the geoprocessing tools. We’ll run this code in a Try statement.  If any errors occur, we’ll capture them in the Except statement and write them to the text file.  We store the geoprocessing tool in a variable to get access to the status code and message.  After the tool completes running, we’ll write two messages to the log file:

  1. The ArcGIS system message which can be confusing to read
  2. A human friendly summary of the tool’s completion
# Run geoprocessing tool.
# If there is an error with the tool, it will break and run the code within the except statement
try:
    # Get the start time of the geoprocessing tool(s)
    starttime = time.clock()

    # SDE is parent geodatabase in replication
    # Change this to your SDE connection
    sde = r"SDE Connection"
    # Child file geodatabase in replication
    # Change this to your file geodatabase
    child_gdb = r"\\path\to\file.gdb"

    # Process: Synchronize Changes
    # Replicates data from parent to child geodatabase
    result = arcpy.SynchronizeChanges_management(sde, "Name of Replication", child_gdb, "FROM_GEODATABASE1_TO_2", "IN_FAVOR_OF_GDB1", "BY_OBJECT", "DO_NOT_RECONCILE")

    # Get the end time of the geoprocessing tool(s)
    finishtime = time.clock()
    # Get the total time to run the geoprocessing tool(s)
    elapsedtime = finishtime - starttime

    # write result messages to log
    # delay writing results until geoprocessing tool gets the completed code
    while result.status < 4:
        time.sleep(0.2)
    # store tool result message in a variable
    resultValue = result.getMessages()
    # write the tool's message to the log file
    logMsg += "completed {}\n".format(str(resultValue))
    # Write a more human readable message to log
    logMsg += "\nSuccessfully ran replication from {} to {} in {} seconds on {}".format(sde, child_gdb, str(elapsedtime), dateToday)

The next part of our script is the Except statements.  I include both the Environment Error and Exception error types.  From what I understand, Environment Errors occur outside of the Python system.  I use it as backup error handling.  The Exception block would handle errors related to the geoproccessing tool.

# If an error occurs running geoprocessing tool(s) capture error and write message
# handle error outside of Python system
except EnvironmentError as e:
   tbE = sys.exc_info()[2]
   # Write the line number the error occured to the log file
   logMsg += "\nFailed at Line {}\n".format(tbE.tb_lineno)
   # Write the error message to the log file
   logMsg += "Error: {}".format(str(e))
   # handle exception error

except Exception as e:
   # Store information about the error
   tbE = sys.exc_info()[2]
   # Write the line number the error occured to the log file
   logMsg += "\nFailed at Line {}".format(tbE.tb_lineno)
   # Write the error message to the log file
   logMsg += "Error: {}".format(e.message)

The final part of the script is the Finally statement.  This is where we will write our general and error statements to the text file.  This is also a good place for other clean-up actions.  Examples of clean-up actions would be deleting MXD variables or cursors to release locks on map documents and layers.

We will open the logFile variable using the with statement.  We will then use the write() method to write the contents of logMsg to the text file.  Using the with statement helps ensure the text file will close, even in the event that an error occurs.

finally:
    # write message to log file
    try:
        with open(logFile, 'w') as f:
            f.write(str(logMsg))
    except:
        pass

Final Thoughts

So that is a summary of the standard script template I use for my GIS scripts that will be set as scheduled tasks.  And to close, here are some things we are automating at Cumberland County:

  • Parent to child data replication to various department file geodatabases
  • Rebuilding address locators and re-publishing geocoding services
  • Rebuilding tiles for cached map services
  • Monthly data updates to a regional group of counties (extract data to geodatabase, zip geodatabase, upload to FTP)

If you’re interested in viewing a recording of the presentation, getting a copy of the slides, viewing the template script, see these links:

Advertisements

One thought on “Automating GIS Workflows with Python Scripts

  1. You may want to consider using try-except blocks for creating a file. Let’s consider a situation where the file is getting created on a network drive. If the network connection is interrupted, the creation of your file will fail and the program will crash.

    Also, if the script is getting run in a read only directory and trys to create a file in the current directory, your script will fail when it tries to create the file also. So unless you know that the file creation will always be successful, it’s not the worst idea to protect your file creation code.

    Great post!

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s