Where On the World Am I?

My daughter had three basketball games this weekend. I am happy to report they won each game by a margin of 20 points or more! A more amazing feat when you consider that my daughter's team is playing against high school freshman and her team has only one freshmen two 8th graders, two seventh graders and five 6th graders! The difference wasn't skill, height or good coaching it was all heart!

Our little school doesn't have a gym to practice in. We practice outside (remember we live in southern California, we've only missed one practice this winter on account of rain.) Without a gym every game is an away game so we have to travel. I've discovered that people have definite differences in the way they communicate driving directions. Some give directions solely relative to street names and turns (left and right). Others use cardinal directions (North, South...) street names and landmarks, and I suppose many other variations and different combinations are also used. I suffer from an unofficial form of dyslexia where I don't know my left from my right! I seriously have to think every time. In my brain left and right are completely relative. I have to mentally perform the pledge of allegiance to know, which is my right hand, that I'm placing over my heart (a genetic defect that I fear I have passed along to my daughter.)

In CAD the universe is a cube, a big Cartesian coordinate system (x', y' and z'). In GIS the world is a coordinate fabric. GIS still stores it's coordinates in a Cartesian coordinate system of x', y', and z'. However, it also includes a definition of the shape of the fabric. The fabric itself has defining geometry. The two parts of the coordinate system the (x',y',z') and the fabric projection, allow GIS to map accurately the spheroid that we live on, and to convert data in one or another flat map approximations of our globe.

When you survey our world and record the coordinates into your survey data collector or into a CAD program you may be accurately measuring, but your encoding of the coordinates is based on an approximation of a flat surface on a round globe. In a small area the approximation is relatively accurate. Over a large area the approximation gets worse toward the edges. The only accurate way to measure on a spheroid would be to use polar coordinates. However this is generally impractical for the work we do.

Think of a triangle. We know from trigonometry class that the sum of the included angles of a triangle always equals 180 degrees. Pick up a schoolroom globe. With your finger draw a triangle from the North Pole down the international dateline to the equator near the Phoenix Islands then across the equator to the 90 W Meridian, near San Cristobol Island, then back up to the North Pole. You notice the sum total of the angles is 90 degrees + 90 degrees + 90 degrees or 270 degrees (not 180!). Were Euclid and Pythagoras wrong, or is this not a triangle?

If you were to survey road features with a local coordinate system to the 10,000th of an inch one square mile at a time, for the entire state of Texas and then wanted to paste those drawings together edge to edge, what do you think would happen when you view the data within the existing State boundary? Are those drawings accurate? Are they precise?

When laying out a large radius highway curve, should it be mapped and stored as an elliptical curve based on its true geometric reality (including the curvature of the earth), or should it be stored with an approximation as a circular curve?

In ArcGIS you can provide the missing coordinate projection information of a CAD file using a companion ESRI projection file to identify which flat map approximation the CAD data was encoded with. ArcGIS looks for projection files when it reads a CAD file. In this way CAD files from one coordinate projection can be accurately viewed with data from other coordinate projections.

How I Build Geoprocessing Tools: #15/15 Deployment

Deployment

Having the right tool for the job is certainly a comforting feeling. Looking at your peg board and seeing only the outline of the tool provokes feelings of dread and frustration and the uncontrollable rhetorical shouting..."WHERE'S MY HAMMER!?" You value your tools, and you like to have them put back where they belong. If you buy a new tool, you want to give it its proper place, in a drawer, on the peg board or in its handy carrying case on a preferred shelf.

ArcGIS geoprocessing tools live in ArcGIS toolboxes. ArcGIS toolboxes can be a .TBX file that lives on the file system, or can they live as a named object in an ArcGIS geodatabase. References to your custom toolboxes can be loaded into ArcToolbox's dockable window interface. By default references to all the ArcGIS geoprocessing system tools are added to the ArcToolbox window. You can load your custom toolboxes that live on the file system or inside a geodatabase into ArcCatalog for easy access.

All Geoprocessing tools in ArcToolbox are automatically included in the ArcGIS geoprocessing framework as named objects along with their parameter definitions and help. Because toolboxes can live in geodatabases you can package your custom tools with the data that they were built to process. Because ArcGIS geoprocessing toolboxes can live in a .TBX file you can easily distribute your toolboxes simply by copying them. Script files are ASCII files that must accompany the toolbox as a separate file. If you have script tools you need to supply the accompanying script files with your toolbox.

I share my sample toolboxes on ArcScripts.com. ArcScripts includes all kinds of useful tools built by ESRI users for all the different software ESRI builds. To share your toolbox on ArcScripts you need an ESRI login. If you don’t have an ESRI user login, follow the instructions on creating a new login on the website. Once you have logged in you can upload your toolbox of tools as a single file or in the case of a script tool you may have to .zip the files.

Back during the original creation of a script tool I always select the store relative paths option when creating script tools in. This ensures that later on when tools are copied the script file can easily be found even when directory structures are different from one computer to the other. I always put the script file in the same directory as the toolbox. This is a good idea for tools that you want to distribute. However you may want to put all your script files in a single directory no matter where the toolboxes are located. You have the flexibility to do which ever is best for you.

To download sample tools use the search engine and its various filtering parameters to find useful solutions. Information is included with each script, listing the programming languages and software the tool was designed to work with. The ArcGIS geoprocessing scripts and ModelBuilder model tools I created would all be considered geoprocessing tools.

Once you download solutions from ESRI’s ArcScripts.com, or copy toolboxes fro your colleagues, you can browse to them using the ArcCatalog data browsing pane. As an extra step to make your tools more accessible you can use the Add option of ArcToolbox to include your new tools in the ArcToolbox interface.

I have posted the solutions discussed in this blog series to ArcScripts. I mentioned that I thought I could come up with another solution to moving data and here is what I came up with. The good news is this solution would support curves; the bad news is it would require an ArcInfo license to run the EXPORT TO CAD system tool. I will use some pseudo code to define this solution and leave it up to you to fill in the SMOP (Simple Mater of Programming).

# GET Input Feature Layer
# GET Output Feature Layer
# GET Translation Coordinates and/or Offsets
# GET the Feature Type of Input Feature Layer
#ADD FIELD - add the field “THICKNESS”
#CALCULATE FIELD – Calculate the Unique Identifier to the “THICKNESS” field
#EXPORT TO CAD- Create a temporary AutoCAD file where the FID (unique identifier is stored in the AutoCAD THICKNESS property. (see December 19th post "Symbology As An Application Messaging System")

#Write an ESRI .WLD file (simple ASCII) of offset coordinates based on user inputs
#GET the appropriate CAD feature class from the newly created AutoCAD file. (The world file will shift the coordinates, including curves)
#COPY FEATURES – copy the CAD Features to the Output FC
#MAKE FEATURE LAYER- create a join-able layer of the new features
#ADD JOIN- Join the Input layer to the Output layer based on ‘THICKNESS’ and ‘FID’ to get back the original attributes of the input feature layer.
#DELETE FIELD- Remove all the unwanted CAD fields from output feature layer

How I Build Geoprocessing Tools: #14 Models and SubModels

Models and Sub-Models

I have taken personality tests in the past, where they have questions like, “would you rather be a concert pianist or a conductor?” Or, some have stated... your watch is a good indicator of your personality. (The first time I heard that I didn’t own a watch!)

I am enjoying very much coaching my daughter’s basketball team. In many ways it is like conducting a symphony. The girls are like the players of the instruments of the orchestra. Each team member learns “the song” on their instrument for their given position. The variations of all the instruments playing the song according to the composer’s orchestration can be combined to make music that is grater than the sum of the parts. The conductor plays the players, which in turn play their instruments.

There is something very organic about ArcGIS geoprocessing models. All because of the intelligent design of the geoprocessing framework, tools can be combined together in intelligent ways to perform a given task. The task that a tool is designed to perform may in turn be useful as part of a more complex task. However each tool can be defined and run separately. If a modification is made to the definition of a tool, that change is automatically realized in each of the tools that has included the tool as part of its own definition.

The ArcGIS geoprocessing framework ensures that once a tool is created it can be used in any of the different geoprocessing interfaces without additional work. This means that my custom tool can be accessed directly from its toolbox with a double-click and a dialog box will be generated to accept the user’s inputs and the tool will be executed. The same is true if I wanted to type the name of my script tool directly into the command line, or include it by name with its parameters, in a geoprocessing script.

I want to use my custom OffsetMove tool in a workflow. I will use ModelBuilder to build a tool that automates this workflow. I create a new model tool by right clicking on a custom toolbox and selecting the New Model option. A new blank ModelBuilder model canvas will be displayed. I can then drag tools from ArcToolbox onto my model builder canvas. I like to use the index tab of ArcToolbox to find tools, because I tend to think about tools by their name. My custom tool is called OffsetMove. By typing it into the search edit box it quickly locates my new custom tool and I can drag it onto the Model Builder canvas.

I will be creating three different tools that will add CAD text to a CAD drawing based on offsets from positions defined by different types of input feature classes. I will create a tool that offsets TEXT from points and writes them to a CAD file. I will also create a tool that creates TEXT in a CAD file based on an offsets from the Mid-Points of linear features, and a third tool that creates TEXT in a CAD file based on offsets from points located inside polygons. Because with certain polygon shapes, like a crescent, the centroid can be outside the polygon, I will select the INSIDE option to force my point for text to be inside the polygon and offset from there. At the heart of each tool is my custom OffsetMove tool.

The difference between my three tools are how I generate the input points that will in turn define the CAD TEXT positions. For line features I’ll use the FEATURE TO POINT tool with the default Mid-Point option, where as for the Polygons I will use the FEATURE TO POINT tool with the INSIDE option. For Point features I can run my OffsetMove tool directly. The part of the workflow that creates CAD TEXT for each option is handled by the ArcInfo EXPORT TO CAD tool. I will create a model that just includes my OffSetMove tool with the EXPORT TO CAD tool, and I will call it PointsToCADText. This named custom model tool will include my custom Script tool OffsetMove tool joined to the system tool EXPORT TO CAD. The result is another custom tool with its own name, and automatically generated interfaces. I will then use this new custom tool as a sub-model in the tool that creates text for lines and polygons.

A sub-model is just another tool. It is added to a ModelBuilder model like any other system or custom tool. Any of the parameters of its included tools can be turned into variables of the sub-model or can be hard coded. The parameters can be renamed and the resulting model tool can have its own help documentation and metadata. It is a tool.

I will expose the X,Y,Z offset parameters as variables as well as the input and output CAD file names in my sub-model. I will hardcode my output feature class, the one I use as input to the EXPORT TO CAD tool, the same that is the output of my OffsetMove tool and define it as intermediate input. It will then be deleted after I run the tool automatically for me. I will also expose as variables some of the parameters of the EXPORT TO CAD tool to provide enough flexibility to create the CAD file.

Sometimes I may want to hardcode all the parameters, or at least provide useful defaults for the various options to make a tool easier to run. If I always choose the same parameters in a model there is no reason to make my custom tool any more complex than it has to be. I will still have the flexibility to change the parameter values of my tool either by editing the model or by exposing the parameters as variables later on and saving it.

I have several custom model tools in my toolbox that are just system tools that I have hard-coded the inputs parameters. These custom model tools may have just a single system tool with all hard coded inputs and outputs and no variables. I just run the tool and it performs its function without prompting.

Continue to Part 15 of 15

How I Build Geoprocessing Tools: #13 Documentation

Documentation

I try to get to the gym at least two times a week as part of my exercise regiment. Every once in a while a new machine arrives on the floor at the gym. Some new torture device with big gears, wires, pulleys, hinges and weights. I assume the device might have some benefit to me, but as I approach it I may not even know how to mount it. That's where those little pictures of people using the device pasted to the side really come in handy. I can do all I can to make a tool as intuitive as possible, but help is always a welcoming and comforting addition to the work since, if nothing else, it confirms our assumptions. It can also warn me against potential dangers of misuse, or compromises I must accept with its use.

I tested my tool with various forms of data, including CAD feature classes as input. Here is where my tool meets the real world of ugly/partially corrupt data, software limitations and compromise. Like I mentioned before, this blog is more reality show than tutorial. Here is the result of my testing: My tool works for POINTS, MULTI-Points, POLYLINE and POLYGONS with the exception of curves. An unfortunate compromise is that the GP scripting tools of ArcGIS 9.1 do not give me access to the curve definitions and therefore the curves are lost when saving to a new feature class. One way to ensure a better approximation with my tool would be to make my output featureclass a shapefile. This is because the first thing my script does is make a copy of the input. If I create an output Shapefile the curves are approximated as part of the copying function. If I specify a geodatabase output featureclass the curves will be retained when copied, but then they will be flattened out when my script modifies the copied features coordinates.

That said, I still have a solid solution for offsetting Points and a partial solution form shifting Polygons and Polylines. I won't bother spending any more blog time with the script itself lets move on...

I want to deploy my script tool. I have the two essential parts of a script, the script file and the GP script tool. They are linked and can communicate one with the other. I am not quite finished however. A good GP tool should have documentation and context help. I add my documentation to the script portion of the tool by selecting the Edit Metadata tool in ArcCatalog that is enabled when I highlight my OffsetMove tool in the data pane, and select the Metadata preview tab. The standard Metadata editor appears and I can type in the supporting help information. This way others can better understand what my tool does and how it works. Inside the script file itself I have added comment lines to define each action in the code and have includes some error trapping and user feedback.





I've renamed my tool to OffsetMove and will put it in my BlogTools toolbox. I will then publish the toolbox on ArcScripts once I add a couple of models that use my tool...

Continue to Part 14

How I Build Geoprocessing Tools: #12 Testing

Testing

We all like to get positive feedback for the work we do. I appreciate those of you who have told me that you have been enjoying this blog. I hope to continue to provide you with useful content in readable format. Negative feedback is equally useful but not as enjoyable. Getting praise and being alerted to my faults helps to direct and finish me. I hope to turn a corner on this series and finish the MOVE tool to distribute to you for your own use and customizing, but we've got a couple more steps...

I finished the first draft of my script, but I have no idea if it works?! I need to test it. I will test the script file within the PythonWin environment rather than launching the script tool inside ArcGIS that would in turn call the script file. I will use the tools within the PythonWin environment to help with any debugging.

As part of my regiment of testing I print out the key variable values within my script to the PythonWin interactive window to get feedback on how my tool is behaving. Later I use the GP.AddError method to write debugging messages to the geoprocessing message window when I'm testing the tool inside ArcGIS. Sometimes these feedback comments are temporary and I remove them before I distribute the sample, other times they are useful feedback to the user while the function is processing, especially for longer running tools it gives some sense of the progress.

Another handy function of the PythonWin interactive window is that once I've loaded the ESRI geoprocesser I can type geoprocessing commands directly into it, and get results. I can test out a single line, or just a couple lines of code by cutting and pasting them into the window.

I will hard-code the input variables directly to string and numeric values with equality statements in thbeginningng of my script and then comment-out the statements that are attempting to get the arguments from a calling function. Hard-coding the inputs for testing will make re-running the function easy since I won't have to enter any inputs. I can change and re-run the script and get feedback instantly. Once the tool is functioning for my test data I can try other data and/or test my script with the inputs coming from the geoprocessing tool, instead of just running the script code directly inside PythonWin.

During testing I will muddle through typographical, syntax and spelling errors... I will probably have a couple of logical errors before its all said and done. Barring any show-stopping design flaws I'll finish with the Simple Matter Of Programming (SMOP).

I will test this function with the original use case: to modify the coordinates of a Point feature class. I originally needed a tool to adjust the position of points to use for defining the insertion points for CAD TEXT entities created using the EXPORT TO CAD tool. I can create CAD TEXT using the optional functionality of the EXPORT TO CAD tool by including a CADType field in that point feature class set to a string value of "TEXT" and valid text values in a field called TxtValue. Since I want the CAD text to be located near, but not on the point features, my MOVE tool will create offset points for this purpose.

Once I've got it working I may add additional code to make it work with any type of features, POINTS, LINES, POLYGONS... I am already thinking how this tool can be repackaged into multiple tools using the same basic logic, but with different interfaces.

Continue to Part 13

How I Build Geoprocessing Tools: #11 Script Parameters

Passing Script Parameters

Perhaps like many of you I dabble in home improvements to save money, and to ruin every weekend for the rest of my life! In December I set a moratorium on projects to be sure to enjoy the Christmas season. Well the doldrums of January are upon us and its back to the honey-do list.

Back in November I installed a three-way light switch to control the lights at the top and bottom of my stairs. Unfortunately, I’ve wired the light three different ways and they’re all wrong. The first way only turned the lights on and off from the top of the stairs, the second way only from the bottom, and its current wired position neither of them are working! (Its been that way all of December.) Getting the information from the switch interface to the light seems a simple task, and it should be, however in this case syntax and wiring are critical or everything stays dark.

The geoprocessing has typed variables. The geoprocessing framework validates its variables according to these types. The Geoprocessing framework creates a dialog box interface on-the-fly according to each variable’s type. When I defined my geoprocessing script tool variables I gave them an order and a type. I use the knowledge of the order and type to build my script code. In this post I will simply explain how I get the information the user enters into the script tool interface, into the script’s python code.

Python is just one of the scripting environments that support the building of geoprocessing tools. As discussed in earlier posts all of ArcGIS geoprocessing is really accessed through the geoprocessor object (GP). One arguable exception to this is how the data is passed to the script from the geoprocessing framework. I use a different programming object to get the parameters into my script. In one of the first lines of my code I asked python to load code to use a programming object called SYS. As far as I can tell SYS means “system”, which I figure is my operating system. I am sure SYS can do a lot of cool things, but right now all I care about is that it gives me access to the variables entered by the user of my script tool. ARGV is a property of the SYS object. It is an array or collection of things, specifically the variables in my scrip tool. Below is a snippet of how I access these values with the knowledge of the variable’s order and meaning in my script.

SYS is a built in module that comes with python. The argv property is a list of the parameters associated with a particular script. sys.argv[0] is always the name of the script and is kind of an "invisible" parameter to users of ArcGIS (although it often comes in handy). Therefore, the SYS.argv list always has a length of at least one. When a script is run from ArcGIS, the application passes parameters to python. In the SYS object, these parameters will start from the [1] index position in the argv list but the list will have a length of 2 because it has that first "self path" parameter. Thus a python script set up to run in ArcGIS where it will have one parameter will have a SYS.argv of length two.

Worth noting is how you might access optional variables based on the number of arguments. Some recomend when dealing with optional variables to put them at the end of the list of variables. However, for style I like to put the output last and other optional variables in the middle. Either way you’ll need a strategy for reading the parameter list you are given from the geoprocessing framework as delivered by the SYS object.

…
# Get the Input Arguments
Variable1 = sys.argv[1]
if len(sys.argv) > 2:
Variable2 = sys.argv[2]
Variable3 = sys.argv[3]
else:
Variable2 = "#"
Variable3 = sys.argv[2]
…

You might assume if you have one optional variable in the middle and that if you have two variables the first is the input the second it the output. Or, if you have three variables that the first is the input the second is the optional variable and the third is the output, etc… Or, you could order the variable with the optional parameters last and just assign the variables in order and if there is an error you’ve run out of parameters and you assign the rest to empty values.

To get variables I use the SYS object. To pass the variables back I use the GP object. I talk back to the geoprocessing framework to populate my geoprocessing tool’s output using the geoprocessor object’s SetParameterAsText method.

…
# Set the Return Output Value
GP.SetParameterAsText(2, OutputString)
…

Note: the parameter index 2, is Zero-based like the SYS.argv except the first parameter in SYS.argv is the name of the calling scirpt tool, so the first variable in SYS.args it one not zero. Therefore the GP.SetParameterAsText variable index is one less than the index for the SYS.argv.

Continue to Part 12

How I Build Geoprocessing Tools: #10 Cursors

Geoprocessing Scripting Cursors

Geoprocessing (GP) scripting is object oriented. With GP scripting you create and manipulate objects. These programming objects have methods and properties. Methods are "actions" the objects know how to perform. Properties are values or other-objects an object contains. Often the method of an object will create and return another object that may in turn have its own methods and properties. A graphic object model map shows how these object hierarchies are built and are related one to another.

A cursor is an object whose primary purpose is to provide access to information from within a database. The GP object's cursor methods create row objects in an orderly fashion. Cursors objects are created for the purpose of accessing data and are transient (temporary). Cursors are "programming things" not necessarily GIS or database "things'. Cursor objects are objects created for the sole purpose of giving you access to your data.

A cursor is a placeholder or a type of dynamic bookmark that allows you to manipulate rows in a database. There are three different types of geoprocessing cursors. I included a section of the scripting object model diagram in the last post that highlights the section that deals with the cursor. On that diagram you can see the three cursors and the things they can do (their supported methods).

All of the cursors give you access to a database row which in turn give you access to a collection of fields and their values. The Search Cursor is a read-only access to a data source. While the Update and Insert cursor also allows you to set the value of a field. In ArcGIS feature layers have a geometry field. Later we will be accessing this geometry field to get coordinates and modify them.

The Search Cursor is good for performing your own forms of selection or reporting. Since the cursor is read-only it is a “safe” way to get information out of a database.

The Insert Cursor will add a row to an existing database. Using an Insert Cursor on a new empty table is how you would build a new database or like its namesake insert data into an existing table or feature class.

The Update Cursor changes existing data. This is the most heavy handed of the cursors because it modifies your data. ...But, if you want to edit data this is the way to do it!

I have a choice then on how we want my MOVE tool to function. Do I want to Update the existing feature class and replace each feature with the modified feature so that the input data is changed? Or, do I want to create a tool that uses the information in an existing feature class to build a new feature class with modified coordinates? The answer is based on the confidence you have on your code. If you are confident that you tool is functioning well then what you may want is to change or edit your input. A more conservative position is to create a new feature class from the input. Especially for testing I would suggest a more conservative approach using the Insert Cursor to create a new feature class rather than the Update Cursor. It would be very simple to modify my script afterwards if Iwant to use an Update rather than Insert Cursor.

The Geoprocessing object has three cursor creating methods: the SearchCursor, InsertCursor, and UpdateCursor methods. After creating the GP object like I did in the previous post. I can create a cursor object by using one of these supported methods of the GP object. Fore example:

#Create a geoprocessing Insert Cursor
MyInCursor = GP.InsertCursor(InputTable)

Where MyInCursor is an Insert Cursor object and InputTable is a valid ArcGIS feature layer or table view.

Continue to Part 11

How I Build Geoprocessing Tools: #9 Pseudo Code

The Value of Pseudo Coding

I love to sketch. Some people read books or watch TV to unwind. I like to draw things, inventions, home improvement designs, etc... The act of drawing forces me to see how things fit together and exposes flaws in my idea when I see it come to life on paper. I sketch with general shapes then add details. I might create a breakout of a detail to visualize a connection or create a small section to scale, on graph or line paper, to get a better concept of form. Creating an outline for a document or pseudo code for a computer application is very similar.

The scripting environment gives me access to all GP tools and allows me to programmaticly assign the parameters of a tool. In that way I can dynamically control all the tool parameters and vary those parameters based on various conditional statements I code in the scripting language. I can call GP tools within a conditional loop or logical branch with more control than currently available in ModelBuilder.

However, the real reason I have chosen to venture into the world of scripting for the creation of this MOVE tool is because I want access to the coordinates of the vertices of features. I am scanning the geoprocessor object model and I see that the coordinates are accessible in the model. Of particular interest is this section that deals with a cursor and geometry.

I'll come back to this diagram often when I need help seeing how things fit together. I will also learn much from how others have assembled solutions before I'm done. Next I like to pseudo code my solution. Pseudo coding is what I call the process of simply writing the comments for functional sections of my code without the actual scripting code. I like to write in outline form what my thought for the flow of the script might be. I then read the comments/code and see if it looks like it will work. I can reorganize the statements and then I just fill in the blanks.

The process of pseudo coding performs two very important functions. First it helps me organize the solution into logical steps and second it acts as the foundation for the scripts comments that are required if anyone else needs to fix or modify my code. I may be very brief with the first draft of my pseudo code and I will add more and more detail as time goes along.

Here is my first draft...

# Create the Geoprocessor

# Get the Inputs

# Create a Cursor

# Get the Geometry

# Modify the Geometry

# Write the Output

My second Draft...

# Create the Geoprocessor

# Get the Inputs

# Create a Cursor

# For Each Feature

# Get the Geometry

# Get Each Vertex

# Calculate the New Coordinates

#Get the next Vertex

# Modify the Geometry

# Write the Output

# Get the next feature

...I continue modifying and adding detail until I'm confident I know how to solve the problem with the tools I have. I will continue to add new sections of detail indenting as I go; first pseudo coding with comments and then filling in the blanks with actual code...

How I Build Geoprocessing Tools: #8 References

ArcGIS Geoprocessing Scripting References

In the last post I created a link to access the geoprocessing scripting object. That object gives me access to all custom and system geoprocessing tools within the geoprocessing framework. When I create a script tool, or model tool or any other form of GP tool it is automatically accessible through scripting using the geoprocessing scripting object. In addition to providing access to every GP tool the geoprocessor also has some special functionality to give you more control over aspects of data elements (like the coordinates), and some additional programming tools.

Here are some of the references* I use to discover what the geoprocessor can do and how to do it. First is the online documentation for GP scripting; the real how to:

Writing Geoprocessing Scripts

Another is an object diagram that shows in a programmer's object model how things are organized.

For all you programmer types the familiar object diagram give you all you need to build your code. It confuses me a little, but if I stare at it long enough I can at least read all the method names and properties and get an idea of what is interesting and possibly useful for my solution.

* These two references are found on ESRI.com and require a user login.

Continue to Part 9

How I Build Geoprocessing Tools: #7 GP Object

Adding the ArcGIS Geoprocessior to My Script

My wife teaches piano, and I marvel how these elementary school kids who months before knew NOTHING about the piano, now decipher a complex scripted symbolic language of DOTS and squiggles on some bizarre line graph into coded note names in some polymorphic key signature, and then translate that information into an audible expression with constantly changing rhythmic frequency using multiple, simultaneous and dynamic patterning sequences with the fingers on both hands and foot pedals to make music. Piano is hard. ArcGIS geoprocessing scripting is easy.

As I delve into the process of writing a script file to define the functionality of my geoprocessing MOVE command I run the risk getting a little technical, but …you wouldn’t be reading a BLOG about GIS and CAD interoperability if you weren’t looking for something technical... right.

In an ArcGIS geoprocessing script tool the scripting tool part I created in the last post communicates with the rest of ArcGIS and also calls the script file that I will start working on now. The script file itself is a simply an ASCII text file. What I put in the text file is a series of comments and instructions that I organize to document and accomplish my desired task. The task I am performing is to take the geometry of input GIS features and create a copy of those features with modified geographic positions. I want to MOVE the data from one location to another.

The syntax for the instructions will be governed by the scripting environment I choose. I will use the PYTHON scripting environment to encode my instructions. Python like all computer programming languages includes functions and parameters, operators and variables, conditional branching, looping, and logical, alpha and numeric expressions. I can perform many useful tasks using this language. However, PYTHON knows nothing about GIS or CAD. The tool I am writing is a GIS tool. I need to teach PYTHON about GIS. All of ArcGIS’s geoprocessing functionality has been packaged into a single functional conduit that PYTHON does understand. This conduit is called the ArcGIS geoprocesser. In my script one of the first instructions I add will make a connection to the ArcGIS geoprocesser. In doing this I will be able to utilize ArcGIS geoprocessing functionality in my PYTHON script. The PYTHON application will evaluate the statements in the script file. When I want to perform some ArcGIS geoprocessing function I will code the request in the PYTHON script using the syntax for the ArcGIS geoprocessor object. PYTHON will pass the request to the ArcGIS geoprocesser application using the object and wait for a reply. Using the two in combination gives me a huge amount of flexibility.

Below is the beginning of my script. It contains some lines of comments and then statements that makes a connection to the ArcGIS geoprocesser. To create the geoprocessor (GP) object in my script I need a valid ArcGIS license, the ArcGIS software accessible to my desktop and the means for the scripting language to communicate with the ArcGIS software. Here is what I do to create a PYTHON script that will use ArcGIS geoprocessing functionally. Below is a script code snippet:

# ArcGIS Geoprocessing MOVE tool
# Written by: Don Kuehne, ESRI 2006
# Description:
# = = = = = =
# This tool will take as input a Feature Layer and offset values
# and output a # Feature Class with a coordinates modified by those # offset values.

# Use the Python ‘IMPORT’ statement to add needed Windows
# communication functionality to this PYTHON Script

import os, sys, win32com.client

# Create a programming ‘Object’ which connects this script to the
# ArcGIS ‘geoprocessor’ using the win32com.client imported above

gp = win32com.client.Dispatch("esriGeoprocessing.GpDispatch.1")

Continue to Part 8

How I Build Geoprocessing Tools: #6 Script Definition

Creating A Script Tool Definition

I recently met with a friend of mine who among many other fascinating accomplishments helped design the first successful human powered aircraft. And, every time I talk with him I am inspired to know that revisiting the same problem with a fresh look, new tools and new methods can an will eventually bring success. People were visualizing man powered flight for thousands of years, all that tried failed to the point that not even motorized flight was thought possible.

(He and I have been collaborating on improving a swimming device to make most anyone, even me, swim faster and farther than an Olympic gold medalist! If you're into scuba... Lets talk!)

Let's visualize and build a script tool. Like I mentioned in the last post a script tool has two distinct parts. The script tool and the script file. The script tool has properties and inputs and outputs that conform to the rules and standards of the ArcGIS geoprocessing framework. The second part is a text file of script instructions, the guts of the tool.

It doesn't really matter which one you create first; the tool definition or the script file. I always like to create the script tool part first since it helps me visualize the inputs and output. Once you create the script tool you can actually use it in a model before you write the script. The tool won't do anything of course, but it will look and act just the same as it will when it is finished, which gives you a great sense of accomplishment even though your work is really just starting.

Let's do it. We need to create a script tool. Tools live in toolboxes. In a previous post we created a toolbox for a Model tool we thought about creating. We can create a new toolbox or just use the one we already created or any other geoprocessing toolbox. Highlighting the toolbox of choice, right-click on the toolbox and select ADD>Script.

A Wizard walks you through the creation of the script tool properties (The wizard is actually the script tool's property sheet presented one tab at a time. After you go through it you can change any or all the values by accessing the tools property sheet). Name your script, and add a description. I always create the tool with a relative path, which makes the tool easier to distribute.

I think the idea is that you normally write your script first and then ADD it, but I don't do it that way. I like to build the script tool interface first and then I come back and add the script. The wizard will create the file if it doesn't exist. (I don't like to see the command window, so I leave the show command window option unchecked).

In this next dialog I am asked to provide my tool parameters in a grid. This is the important part. Geoprocessing tools have typed parameters. This typing allows the geoprocessing framework to provide validation and to understand how tools can and should fit together. Without this information, the geoprocessing framework will not understand how this tool relates to other tools in geoprocessing.

The order of the parameters in the grid determines the organization of the parameters in the dialog, on the command line, or in another script. The data type is very important because this determines how the tool will function in relation to other GP tools. Also critical are the individual parameter properties set in the lower grid for the highlighted parameter. In this lower grid the parameter's properties are set that determine if the value is required, is input or output, or if it has dependencies or constraints.

My first parameter will be the Input Features. Right away I have an important decision to make. What type of data will I accept as input? There is a subtle difference between a Feature Class and a Feature Layer. For most tools you should use a Feature Layer since it would include layers built from Feature Classes that have selection sets and joined tables, whereas the Feature Class choice is limited to the data in that feature class as it sits on the disk. The same is true for the distinction between a Table and a Table View.

I then can go ahead and enter my other parameters:
X', Y' and Z' offset parameters, which will be Optional inputs of the type Double without any other restrictions. The Output features will be a Required Output feature class,
(which means my script will not run unless an output path to a feature class is provided. It is up to my script to actually create the output. )

Presto, I have created a finished script tool (definition). I can double-click the tool in my toolbox and the tool's dialog box interface is generated on-the-fly based on the tools parameter properties, according to the types and order of the parameters I specified. The geoprocessing framework is now able to use my tool. It is my responsibility to make-good on my promise in the script file to create the correct output, but as far as the geoprocessing framework is concerned I have provided enough information to use the tool.

Let prove the point by adding a new model tool to our toolbox and use my MOVE tool (definition). We can drag my new custom GP Move script tool on to the ModelBuilder canvas. When I double-click the tool you see a dialog interface is generated on-the-fly for my MOVE tool based on the parameters I defined in the script tools property sheet. You will also notice the parameter types and an output bubble are automatically generated according to these properties in ModelBuilder. When I browse for input data from my tool the tool automatically filters the data for valid inputs. I can actually populate the parameters for the tool and see the colored validation symbology of ModelBuilder functioning on my unfinished custom Move tool. Granted it won't do anything if I run it, but we're making progress!

I want to reflect a little on what just happened. The entire user interface (UI) for my tool was created on-the-fly based on properties I entered into the script tool property sheet. This tool was integrated into the geoprocessing framework automatically. All validation for the inputs and outputs were done for me automatically. ModelBuilder is awesome and we'll see lots more of it before this is all done, but let's give the designers of GP scripting some credit, what I just did was pretty cool!

Continue to Part 7

How I Build Geoprocessing Tools: #5 Scripting

Scripting and Other Not So Scary Things

My daughter is in seventh grade. She is enrolled in a university model school and attends classes twice a week. She is taught in a normal classroom setting, but all of her work is performed at home, like you would when attending university classes. At home my wife and I become her teachers. In that way her school is a hybrid private school/ home school. She has choices of which electives or independent courses of study she wants to take. Instead of enrolling her in the math classes offered at the school we decided I would teach her math.

My teaching style for math is to present simple concepts and overly complex homework to show how the complex application of a problem can be broken down into simple steps. I remember being confused in elementary school by simple repetitive homework because I thought I only learned a small thing when in fact I really knew more. For example, when I taught my daughter two digit addition in 1st grade her homework consisted of 15 digit number addition problems. It’s the same technique just more carrying of the “ones” to the next column. Likewise, subtraction, division, multiplication, etc… unfortunately/fortunately she ended up learning most of 1st – 4th grade math the first month or so. Now, she really likes math because I think she feels empowered (or, despite her teacher she’s really smart). I hated the repetition of elementary math, one reason why I like computing; if I do my job right I only have to solve any given problem once. Like you, I want to solve problems not calculate pages and pages of long division.

I chose the MOVE data tool example because I had a pretty good hunch it was something that wasn’t going to be straight forward to accomplish in ModelBuilder with system tools alone. The task is easy in ArcMap with the standard user interface, and in ArcObjects it would be an elementary task for someone with those skills.

I still think with a little compromise and bending I could come up with a ModelBuilder solution using system tools only, but all signs seem to be pointing to the creation of a geoprocessing script tool for this BLOG series.

So it appears I will begin my sample geoprocessing MOVE tool with a little more complexity, and then we’ll put it all back together and hopefully make it look easy in the end. I think the fear of a scary “looking” math problem or a new computing environment can best be handled by embracing the scary parts in context. At first I myself avoided geoprocessing scripting all together with my realization that I was not a programmer. I soon found that scripting often simplified tasks rather than making them more complex. And, I still didn’t have to become a programmer.

For those of you hoping to get a better look at ModelBuilder stay tuned, we’ll be coming back to ModelBuilder with our new script tool to automate a couple of different use cases.

The actual script code of a geoprocessing script tool can be created using a number of different development tools. Without wading through all the details, Python is the best choice for my application because it comes with the software and is more powerful than the others.

The ArcGIS geoprocessing environment treats an ArcGIS system tool the same way it treats a custom Model tool or a custom Script tool. The application and user experience are the same no matter how a tool was crafted. So once we build this tool we will use it the same way we would use any other tool.

In my last post I mentioned an email I got from a colleague that will help me with a lot of the guts for this solution. (More about that later.) Before that we’ll take a look at the two parts of a script tool and how to build them. First is the GP script tool, which is like a container that defines the types and names of any inputs or outputs and then points to the actual script file. The script tool has properties common to all GP tools and lives in a toolbox. The script tool points to a file which contains the instructions the tool will perform on the given inputs. The script itself is comprised of lines of ASCII text stored in a text file that are interpreted by the chosen scripting language.

In the next post we’ll build the actual script tool part…

Continue to Part 6

How I Build Geoprocessing Tools: #4 Storyboard

Storyboarding

As promised I am documenting the process I use to build geoprocessing sample tools. As a disclaimer I am not a programmer or the son of a programmer! I do create sample tools and in-so-much-as you find this useful please continue reading. This is more reality show than tutorial.

Last post I did some investigation into the standard system tools to discover how I might paste some tools together to serve my needs to create a MOVE data tool I can use in ArcGIS geoprocessing. What I've come up with, (and you may have discovered your own solutions and are welcome to add them to the comments below) are a couple more possible tools I might be able to use. I haven't actually thought it all the way through so enough words, lets just try it.

One nice thing about ModelBuilder is that it is a visual programming tool where the interface is generated during the process of assembling the solution and the application is a flow chart or storyboard. I like to build GP tools inside of ArcCatalog rather than ArcMap since it has the best access to data, to preview and browse.

Geoprocessing tools live in toolboxes. To create a toolbox right-click on a folder in the data tree browser pane select New>Toolbox. At this point you can highlight and rename the toolbox or right-click and hit properties to rename it with a dialog.

To create a new model tool I select my toolbox in the data pane, right-click and select New> Model, which invokes the ModelBuilder application.

I like to drag system tools out of ArcToolbox onto the ModelBuilder canvas using the ArcToolbox Index tab. I find it easier to navigate to a GP tool by name, rather than by category because of my familiarity with the tools. Therefore, I use the intelli-sense on the Index tab of the ArcToolbox application pane to highlight the tool and then drag it to the ModelBuilder canvas. I quickly throw together the tools I think might work for me and start moving them around anconnectingng them up.

As I'm putting these system tools on the ModelBuilder canvas I notice a couple of things, some of the tools I've specified require an ArcInfo license, some only work on coverages, (not necessarily a problem since I can easily convert), but ... Hmm... maybe GP scripting may be looking like a better solution? ...But I will press on in ModelBuilder!

Looking under the sample toolbox that ships with ArcGIS 9.1 I also see some interesting sample tools built by ESRI. I see the tools: Write Features to Text File and Create Features From Text File, which is interesting since unlike GENERATE they work on feature classes with ArcView . I'll throw them on the ModelBuilder canvas too and see if any of this takes me where I need to go.

Ultimately I want to specify an X', Y' and Z' coordinate offset for an input feature class and get an output feature class with adjusted coordinates. It would be nice if all the attributes came along too. Looking at the help for each tool by double-clicking each tool in ModelBuilder I read in the tool help that both the ArcInfo generate-Ungenerate and the other GP sample tools don't carry along the tabular attributes. Furthermore, the ADD XY and CALCULATE tool combination would give me the coordinates as field attributes, but with no straightforward way to get those coordinates into a TEXT file that either GENERATE or CREATE FEATURES FROM TEXT FILE could read.

I received an email response from my inquery about existing technology to create a MOVE tool from a colleague here at ESRI today that may hold a more promising path... stay tuned for more... GIS and CAD Interoperability Unscripted!

Continue to Part 5

How I Build Geoprocessing Tools: #3 Scope

Needs/ Scope / Research

Building a geoprocessing workflow solution can be like yunk yard wars, where you build your solution from parts of other solutions you find laying around. Looking at a tool for its primary and alternate uses helps you build custom solutions will less effort.

In my previous posts I had mentioned that I will be documenting the process I go through in creating a sample geoprocessing (GP) tool. I will take a step back then and discuss the step of deciding what tool I need. By considering specific work flows and general use cases I have decided on the tool I will build next. Working for a software company my primary directive is to help others identify and improve their workflows. Users of GIS software are diverse and therefore a tool that I create should be of general use, or serve as an example or building block that can be used in their custom workflow solutions. Sometimes sharing a specific and involved workflow solution can help demonstrate what you could do, but might not be directly useful for your specific purposes without significant modification. Here are some of the thoughts I had when I was deciding what tool I would build for this series of BLOG posts.

1. I need simple GP tool that won't take too long to build.

2. I need a GP tool that can be understood by a large audience of people.

3. I need a tool GP that will be useful since part of my job is to create usable samples.

4. I need to place CAD TEXT more specifically in CAD drawings using geoprocessing. I can create various useful points from lines and polygons; vertices, centroids and midpoints using the GP tool FEATURE TO POINT. By including the CAD field "CADType" in the EXPORT TO CAD tool to override the default creation of a CAD points from GIS points I can create TEXT entities from the any GIS points. A GP tool that can create an offset points from original points would be a way I could place CAD TEXT relative to the points in ArcGIS with a ModelBuilder model that included the EXPORT TO CAD tool.

5. In ArcGIS I may need to nudge an entire dataset to help move it into a corrected coordinate space, or to adjust for CAD symbol placement or CAD TEXT offset errors.

6. I can think of a number of other reasons you might need to move data using a geoprocessing command.

With these specific and general requirements, my next step will be to search the existing ArcGIS system toolbox for tools that might perform this task already or that can be used to build a solution. There may be a tool that performs this task as a sub-function, or maybe there might be a combination of tools to perform our task. Sometimes a powerful system tool might have advanced options or sometimes you could dumb down a tool with trivial or null inputs to perform a different function. A quick look at the ArcGIS 9.1 system toolbox (Online Quick Reference Guide) results in some tools that at first glance I think might have some relevance.

FEATURE TO POINT: Although it creates points from other geometry there is no way to control an offset. We'll keep it in mind.

NEAR: No direct help, this tool can determine proximity and even measure and offset distance, but not create points.

ADD XY: This tool is interesting because it can read coordinates and place them into field values. That doesn't directly help me unless I can get the coordinates modified and put into an output feature class or use it to modify the existing coordinates.

GENERATE: This tool will create an ArcInfo Coverage (old school) from a text file. Interesting perhaps...

CALCULATE FIELD: This tool can modify numeric field values, hmmm... if I have the coordinate values in fields I can add an offset to them using this tool. We could generate a text file from modified field values created by ADD XY and CALCULATE FIELD and create a coverage that we could convert back to a feature class.

Using the POINT feature class as an example all I would need is a way to create POINTS from a table. I'm pretty sure I've seen sample code to do this besides the GENERATE (coverage) tool, but what about POLYLINES and POLYGONS?... In my requirements I've talked about a workflow for CAD text placement with POINTS so a very specific tool to perform offsets with POINTS could utilize the FEATURE TO POINT, ADD XY and CALCULATE tools with a form of GENERATE points to establish POINTS offset from some original points that I EXPORT TO CAD. I'll do some more research and see if I can find what I need to create a tool that works for all feature classes... before I cut functionality.

I decided that the scope of work for my tool will be created in ModelBuilder if possible. If it requires some more power than at the very most I'll create a Python script to help me. While searching the ESRI EDN site I did find an ArcObjects sample that moves features, but I will hold off using it for this example (However you may still want to use it).

I will continue to search the ESRI EDN and ArcScript site for custom GP tools people may have already assembled. I will send out an email to my colleagues to see if they may have already built something I could use. Let see what we come up with...

Continue to Part 4