I have a Ghost in My Teamcenter System

Back in September 2012, one of our clients called me and said, “I think we have a ghost lurking in our Teamcenter installation”. His user was complaining that he had lost a weeks worth of work when the assembly he was working on suddenly vanished from the folder where he had copied it. Not only that, but the assembly was nowhere to be found in the entire database. My client wanted me to investigate this scenario and find the root cause and fix it. My first thought was that the user accidentally deleted the assembly, and is now blaming it on the mysterious “ghost”. We chalked up that incident to an accident. A few weeks later I received another call from a different client describing the exact same scenario. This blog is about my investigation into the root-cause of this ghostly behavior.

The Story

This was a Teamcenter 8.1 installation.

There were five engineers working on an engineering change. They all belonged to the Engineering group. Each of them had created their own project folder while working on this change. In that folder they had copied assemblies and items that they needed to work on. Some of those assemblies and items were created by them, while some were created by their fellow teammates. As such, they were the individual owning users of the assemblies and items that they themselves had created. However, Engineering was the owning group of those assemblies and items.

One fine morning, one engineer found that the assembly he was working on the last few weeks was gone from his project folder. The assembly was created by his fellow teammate, but the engineer had copied it into his own project folder. Upon checking around, all of them found that the assembly was gone from their individual project folder too. But to make matters worse, they could not find the assembly anywhere in the database. The assembly had simply vanished from the database, as if it never existed.

Armed with this information, it was my job to hunt down this ghost.

Setting the Stage for the Trap

After working through a few scenarios and going back and forth, I finally came up with the exact use case to demonstrate this behavior. What I found really surprised me.

Test Assembly, Components & Folders

Here is a snapshot of the assemblies and components created by engineer-1 (Fegade, Yogesh), and engineer-2 (Knab, Don).

Data Structure, Ownership & Protections

Notice a few things from this figure and analyze them to see if there is anything obvious in this setup that is out of the ordinary:

• “yfegade” is logged into the Teamcenter session in the Engineering group.
• The owning user of the items, assemblies, and other data are the engineers themselves, but the owning group is the Engineering group.
• All four items are copied into the YF-ToDeleteOrNotToDelete folder owned by “yfegade”, as well as in the DK-ToDeleteOrNotToDelete folder owned by “dknab”. Users copy objects into their respective folders (be that a Home folder, Newstuff folder, or folder like YF-ToDeleteOrNotToDelete) all the time, so they don’t have to search for them every time they want to work on them. This is especially true when there is a large team working on one project or change.
• Two components (YF-COMP-001 and DK-COMP-002) are used in two assemblies (YF-ASSY-001 and DK-ASSY-002).
• Data is not released.
• Courtesy of the out-of-the-box “Working” Access Manager ACL, “yfegade” (being the owner) has READ, WRITE, and DELETE permissions to the objects that he owns. Nothing out of the ordinary here, as this is the out-of-the-box ACL.
• Courtesy of the out-of-the-box “Working” Access Manager ACL, “yfegade” (as a member of the owning group) has READ and WRITE (but not DELETE) permissions to the objects that are owned by “dknab”.
• Although you can’t see it from the image, the two assemblies YF-ASSY-001 and DK-ASSY-002 are not used in any other assemblies…yet. This is also not unusual. I have seen users do most of their work on the assemblies and components in isolation prior to using them in any other assemblies. Once their work is done, they add/substitute them in other assemblies.
• All this means that “yfegade” can edit objects owned by “dknab” but can’t delete them. Remember this statement; I will come back to this later. This is not unusual either. In-fact this is desired in most situations.

The Test

I tried to delete DK-COMP-002 from YF-ToDeleteOrNotToDelete folder. It failed because I don’t have DELETE Access to it, as it is owned by “dknab”. Similarly deleting DK-ASSY-002 from that folder failed for the same reasons. Next I tried to delete YF-COMP-001 from that folder. It failed too. I have DELETE Access to it, but it is used in other assemblies.

But when I tried to delete YF-ASSY-001 from that folder, Teamcenter happily deleted it. That, I did not expect. The item YF-ASSY-001 was referenced in two folders YF-ToDeleteOrNotToDelete and DK-ToDeleteOrNotToDelete. I searched the database for that item, but it was nowhere to be found. Sure enough, the YF-ASSY-001 was gone from the DK-ToDeleteOrNotToDelete folder.

How can Teamcenter delete an item that is referenced in another folder? After all, if an object is being referenced in multiple folders, then that object can’t be deleted until all such references are manually removed. Didn’t Teamcenter Engineering (and iMAN) work like that forever?

Even the manual says so. I checked the TcEng2005SR1, TC2007.2, TC8.3, and TC9.1 manuals. Look for “Getting Started with Teamcenter” manual, chapter 4 (“Common Teamcenter Tasks”):

V9.1 Getting Started With Teamcenter, Chapter 4

So, what happened and what changed? The answer is in chapter 4 (“Common Teamcenter Tasks”) of the TC10.1 “Getting Started with Teamcenter” manual. However, this update in the TC10.1 manual is about 3 Teamcenter versions too late.

V10.1 Getting Started With Teamcenter, Chapter 4

This change in delete behavior was introduced in TC2007.x (I could not verify if it behaved like this in TcEng2005SR1. If anyone has any updates please share). There was no documentation or release bulletin about this change in delete behavior. This change in delete functionality was so fundamental that I would have expected some kind of awareness or publicity campaign from Siemens. May be that was a little too much to expect.

I had finally hunted down the Ghost!!

The Ghost in the System

As a part of improving the user experience, Siemens had quietly changed the delete functionality. If permissions are right, and if the object is not referenced by any non-folder objects, then delete quietly cut the object from all referencing folders and then deleted it. When I say permissions are right, it means:

• The user deleting the object must have DELETE permission to that object itself, and
• The user must have WRITE permission to all folders in which the object is referenced (so the cut operation succeeds).

Remember the sentence earlier that I said I will come back to: “… this means that “yfegade” can edit objects owned by “dknab” but can’t delete them”. That was the Ghost. I could edit (had WRITE access) the DK-ToDeleteOrNotToDelete folder. Hence I could remove objects from that folder, and that’s why the delete operation succeeded when I tried to delete the item YF-ASSY-001. I had DELETE access to the item and I had WRITE access to all folders in which this item was referenced.

The Impact

The real problem was that there was no warning, indication, or acknowledgement dialog that the object being deleted is referenced in multiple folders, and will be removed from those folders before deletion. There was no such indication to the user deleting the object, and there certainly was no indication to the owners of all those folders where the object was referenced. One can easily lose an unreleased top level assembly or an unreleased engineering change object causing significant loss of work.

If I’m alerted with a warning first, I may be careful about deleting an object. I may do due diligence on my part before I hit that delete button. But that does not prevent others from deleting objects that they own but are referenced in my folders while I am working on those objects. I guess what I am saying is that:

If I shoot myself in the foot, I am an idiot. But why do I have to allow others to shoot me in my foot?

Calling the Ghostbusters

The next challenge was to figure out how to prevent this from happening, or in other words how to restore legacy behavior. There were a few options, but in the end only one worked. I will discuss all of them here.

Call GTAC

This was the first and obvious choice. However, they closed my IR stating that the behavior was as designed. They offered me to convert that IR into an ER….but we all know how that works. I most certainly don’t blame GTAC for closing my IR.

However, The GTAC team and the product development team later confirmed that this change in behavior was the result of a request from a customer trying to prevent their users from manually removing the object from each referencing folder prior to deleting it.

I certainly understand this argument. However, I would at-least expect a preference (or some other mechanism) to control this new behavior (i.e. use new behavior or keep legacy behavior). Without any visibility into this change, just as some customers benefited from this new behavior, some customers lost many weeks worth of valuable work.

Add Custom Pre-Condition to the Delete Operation

My next attempt was to add a custom pre-condition to the TC_delete_msg message against a WorkspaceObject (WorkspaceObject, because that way I could catch not just an item, but folders, forms, datasets, etc.). I would program it in such a way that this pre-condition would call WSOM_where_referenced to find the first level references which are folders. If the count is greater than zero, I could return an error code preventing the entire delete operation on the object. If zero, return ITK_ok and let Teamcenter do its magic to delete the object. Granted, this option does not give me the folder context in which the user has selected the object, but it’s the least impact solution because of its complete transparency to users.

I thought that should be pretty simple. But I was wrong!!

The TC_delete_msg message is implemented only on Items, ItemRevisions, and Forms. Had this message been implemented on WorkspaceObject, it would have covered WorkspaceObject and all its descendants. Unfortunately this message is not implemented on WorkspaceObject. Then I thought, at least I could implement it on Items, ItemRevisions and Forms. That should protect my customer from the majority of mishaps, if not all.

But I was wrong again!!!!

The pre-condition method against TC_delete_msg on Item, ItemRevision, and Form registered fine. But inside the pre-condition method, WSOM_where_referenced returned a reference count of ZERO. After analyzing the journal file, it turned out that the portal (rich client) code was removing the object from all referencing folders prior to passing control over to the server to execute TC_delete_msg message. So it was no surprise that the server could not find any folder references.

Why it was implemented like this, I don’t know. I have always advocated implementing only UI specific logic in client code while implementing all business logic on the server. That way any client (portal, web, SOA, standalone ITK program, etc.) can access the same business logic without having to replicate it (possibly in a different programming language) across multiple clients. But, that’s a topic for another discussion.

Combination of Client & Server Side Customization

I had no choice but to implement a combination client/server side customization to completely control the delete operation, or rather bring back the legacy delete behavior.

• I had to hide the out-of-the-box “Edit -> Delete” option using menu entry suppression.
• Implement my custom “Delete” option on customer’s custom menu.
• Write custom a user service to check for any folder references. If there are any folder references other than the current UI selection context, return an error code, else return ITK_ok.
• Inside the custom delete operation, call that user service. If the user service throws no exception, execute the TCComponent.delete () method on the object, and let Teamcenter do its magic to delete the object.

This was the only solution that seemed to work. But I feel it’s the solution that I shouldn’t have to code to fix something that should have worked to begin with.

Other solutions discussed

There were some other solutions thrown at this problem. I will mention a few:

• Allow only the owning user to have DELETE access: Well this is not a solution. That’s how it works using the out-of-the-box access manager “Working” ACL.
• Allow only the DBA or Group Admin to have DELETE access: This will work, but it will certainly put an undue burden on the DBAs or the Group Admins to do such a mundane task.
• Allow only owning users to have WRITE access to their folders: This will certainly work as a solution to the problem. However, at the same time it will also limit the team’s ability to work as a group. Many project teams organize their objects in some hierarchy of project folders. This solution will certainly put undue burden on the owning user of those folders to constantly add/remove objects to those folders.
• Explicitly checkout the objects that you are working on: This would work too, but is impractical because of the overhead and maintenance required to remember to checkout an object while using, and remembering to check it back in after the work is done.

What Now

I finally raised this issue via the PLM World forum and then to the Siemens Product Development team. They agreed to implement a mechanism (preference or otherwise) to allow users to control if they want the new delete behavior, or keep the legacy behavior.

As per my last information, this new mechanism is supposed to come out in TC9.1.2.6 and TC10.1.1. I have not tested if these versions include such a mechanism or what that mechanism is. If anyone out there has done some testing on these versions, please share your findings. Your posts will most certainly be appreciated.

The Ghost Struck Again

While I was pushing this subject on PLM World, I had one more case of ghostly behavior reported to me:

• One of my users complained that editing the ECN Form deleted the ECN itself. This user was working on an ECN for a month. He had collected some 8 affected assemblies and about 60+ solution items. He was just a couple days short of pushing this ECN through the release workflow. He opened the ECN form to edit some data and Teamcenter replied with an error… something like – Unable to checkout, invalid object. He restarted his Teamcenter session only to find out that the ECN was simply gone from his folder and the database.

Knowing what I know now, I can certainly relate to this story from one of the users from a long time back. He complained that the release_man utility had deleted the item revision rather than quick releasing it.

Final Thoughts

While this new delete behavior is certainly beneficial to some Siemens PLM customers, others have lost significant work. They don’t have a choice but to either accept it as a new way of deleting objects (with the potential of accidental data loss) or employ some workaround to restore legacy delete behavior.

Functionality as critical as delete, which by the very nature of being irreversible in Teamcenter, should have all kinds of safe guards. I would rather go the extra length to remove references from all folders, than having them removed automatically not knowing what those references were.

As far as SiOM Systems is concerned, we follow these guidelines when we develop our code/products:

• Don’t pull the rug out from under your customer’s feet. I.e. don’t change existing functionality without telling your customers.
• If you must change existing functionality, inform your customers. But keep existing functionality as the default and employ some mechanism to switch to the new functionality.

Further Information

For those interested in reading the original thread on PLM World, please visit the Original PLM World Thread.

Acknowledgement

I would like to thank Jon Jarrett from ATK. He assistance was instrumental in raising this issue to Siemens PLM Product Development Group.

Next Steps

We would certainly appreciate your comments on this topic. Particularly we welcome your opinions for and against this new delete behavior. I think the reading community will benefit from such comments.

We also encourage you to explore our website to learn about the products and services we offer. If you have questions or wish to discuss this topic or any other topic with us in private, please contact us.