My ego blew up today.
Complete cranked out a Motoman 3 Robot Cell from an Absolute trash state that “ran auto” at 80% speed to an optimized auto at 100% without a single issue on the robot side. It basically can't run auto for shit when I started this Tuesday. It won't achieve cycle time, and it clips tooling everywhere, the only process which ran auto was the normal welding process, none of the other decisions or processes are done, and some reject/red rabbit paths are taught but not optimal. Job structures had to be changed and completely redone. The project manager came down to me and started having a meltdown saying he thought it would be a quick job and started jabbing people for “standing around and doing nothing” and “Some people *take* their fucking time”
Guess what. I've done my fair share of fucking around, but if we don't have the right person for the right job, or the fucking equipment to be able to do the job, I'm just going to fuck around.
Who worked on this machine before me and got it to “Ran Auto”? 2 of the most experienced contractors at my company.
Yea, my company really like to think people are replaceable and just throw people onto different project because they got years and years of experience with Fanuc robots.
Too bad, not like a PLC programmer. Robot experience does not fully transfer to different brands of robots. You throw people who are not experienced on a specific Robot, they are not going to get it done properly.
Well, thanks to my company I know exactly how to deal with all the issues that come with Motoman robots, from safety, and general IO to robot motion. I've worked with them for 2 & half years now. I've been in the company for 3 + 1.5 yr as a paid intern. I knew nothing about Motoman before I started working there.
Here's what I've done in 30 hours ~ 3 work days:
I perfected the existing program for the 2 Robots that interacted with the Fixtures on one side of the Turntable. Meaning, paths are completely re-taught – optimizing their air moves from a “Fine linear moves in the air everywhere” kind of shitty state into a state where motion can not be more smooth (Smallest Overall Joint Movement possible to reach all process positions while avoiding obstacles). I explained once or twice to everyone who worked on this project how to teach or optimize the robot motions to deal with the issues that come with Motoman Robots – Large TCP Reorientation during linear Moves does not stay on that absolute linear path, the TCP will wander around as if it isn't a linear move – on any other commonly used robots, for example, Fanuc, Kuka or ABB, with the correct package – usually come standard – will always stay on the linear path if it's a linear move.
So I re-taught and optimized the program on the one side that has been “Running Auto”. Because it is identical to the other side, I basically copied the programs and performed some touch up – due to the physical imperfection of the fabrication process. Knowing that Motoman robots will move a bit more aggressively at 100% speed vs 99% speed, I had concerns I might have a few crashes since I drastically changed how the robot moves from when it “Ran Auto” before.
What else needed to be done when you change robot motions? Check for interference between other robots that also interact with the same process area. This usually takes excruciatingly long depending on how many robots and how many variations of the processes are present. The amount of work needed to be done here is not linear. Say, it takes 1 hour for 2 robots, and if it's 4 robots it would take 4 hours? No, this works like O(logN) – look up “BIG O NOTATION FOR COMPLEXITY”. The more robots interact with the same thing, the more degrees of complexity it adds, which adds one variable to the probability. Ofc, the reality is, if a robot is completely outside of another robot's reach, it can't possibly have any point of interference. So this is really a grey area when it comes to estimating. But if you saw my “Smallest Overall Joint Movement Possible” statement from a minute ago, you'll know the less the robot moves, the less possibility for interference. Which is why I always ask people to optimize their robot motions, most people don't give a shit though. Especially people who don't get paid enough to care.
This time, it's not that bad. And because the 2 fixtures are identical, I zone it once for one side, the other side is also theoretically done. Doing this correctly even with just 2 robots still takes time and testing – as if not done correctly, robots will crash. On top of this, we build automation equipment not to show off. We need to hit a “Cycle time” target. And if 1 of the robot waits on the other robot for an interference zone for too long, we can kiss cycle time goodbye. So I made sure whenever the 2 robots can move at the same time without touching each other, they will move at the same time and not touch each other. It involves entering and exiting 2 different zones recurringly throughout the program. One of the zones, I realized after we started auto-testing, has to always be entered first by one of the robots in order to not cause a “Race condition” causing the other robot to wait until the robot in question finishes its process and then starts to move in – causing a big delay in the cycle.
When I started, it was these more experienced people who taught me this stuff. I appreciate my learnings very much. But I despise not taking time and aiming for perfection, which most people with experience know exactly where they can slack off and not get in trouble. I don't blame them. No one is paid enough for bullshit, so everyone will take it easy when they can.
By design, this machine's main process on each side of the turn-table also had 2 stages that can individually be processed, and while the second stage finishes first (yes the second stage in this process is processed first), a 3rd Material Handler robot must be able to pick the second stage without interfering with the 2 robots processing the first stage. It is possible but I needed the tooling to be absolutely clear after the Material Handler picks in order to continue the first 2 robot's first stage. Luckily first stage process is largely out of the way.
Had to touch up the pick position for the second side fixture just like I did for the process programs on the first 2 robots.
Furthermore, Motoman robots have a weird issue when it comes to Spot Welding Servo Guns mounted as their manipulator. If a carried spot welder gun is equipped on these robots, during motion at 100% auto speed, the spot welder servo gun will not open fast enough for it to clear the tooling before the robot Arm tries to go to the next position. There is a fix, and it is simply adding a “WAIT SEC 0.01” after the move where the gun opens to fix – and contrary to most robot technician's belief, this does not cause motion rounding to become a FINE movement, Motoman robot will move as smooth as if it weren't there, and only allow the servo gun to open more before proceeding to next move. (There is no setting to change this behavior on a Motoman Robot – AFAIK and as a Motoman Specialist would know. I checked hundreds of “System parameters” for a setting like that and come up with nothing). Nobody in my company did fuck about it either. Some came to me after explaining that they crashed when they rose the robot speed from 95% to 100% and it crashed – I don't know what to say, because I've explained this could happen multiple times and even wrote it done in documents as per my company's “Continuous Improvement” program, and I personally warned him about the issue. I don't blame him, good dude, doesn't fuck around, and gets shit done, just didn't work with Motoman a lot, and probably assumed it's just gonna be like any other robot.
I also had to add program logic to allow the robot to execute programs other than the main process – Rejects, Re-Pick then Reject, the vast majority of the paths are taught, but I had to optimize a little and double-check everything just to be sure. I re-structured the programs to incorporate every scenario possible and created VIA motions to make sure the robot could go back and forth from the end of most of the programs to the beginning of most of the others.
Long and behold, our Project Manager came to the shop floor and voiced his displeasure of having to talk to the customer about the situation that I had to spend more time getting this thing to run auto when he was told the machine “Ran Auto” before and we've done a lot of what we can do to make what still needed to be done, a “quick job”.
(I took Monday off as a sick day because I physically felt terrible after waking up late to work already – probably because I assembled a makeshift wooden table out of kitchen cabinet panels on Sunday within an hour using hand tools, a drill bit, and 12 screws – which I've never done before. It worked great for what I needed, but not my back.)
Guess what, I was busy with other projects, or out of town, and had no idea what the status of this project is – everybody is busy and had more pressed projects to work on, plus I expected better from our more experienced, more expensive contractors. I didn't even know about the timeline – my methodology of working fast is to strike perfection, once and do repetitive tasks effectively. Did I mention I wrote a whole bunch of Python programs at work to analyze Motoman Robot Backups, fix simulation errors that prevent me from loading simulation programs to actual robots or extract/format all data from a WTC welder backup report, and because changing IP takes too long, I wrote a Batch Script to change my IP addresses with support for customized shortcut IP lists within 5 seconds each time and not needing to touch mice? And there are people complaining to me that our company laptop had an IP configuration tool that does exactly the same thing. Well, too bad. Fill me in earlier before I make my own solution while I hurry up and wait.
Speaking of WTC welders, the previous machine run we had didn't use weld schedules the way it should've been used as a production-ready machine – again, it was rushed. I had to change the weld schedules on 3 WTCs according to the supposed schedules based on quality reports from before, and also adjust the robot programs for new schedule calls.
Anyway, back to the robots,
I had to slow down some moves due to the robot clipping safety boxes(yea Motoman robots predict themself in/out of any safety boxes, even if they are not in/out of there yet). I had to move the logic handshakes with the PLC a few times to improve cycle time or prevent possible lock-up scenarios. Again, I rushed all these tasks while aiming for perfection, there's gotta be some mistakes. Good thing is, I only left the mistakes in if I know I could catch them later during debugging.
Here we are, 30 min before the end of the day today, I ramped the robot from 65% speed straight to 100% speed, concerned about whether the robot' is going to start crashing again as how it went in the past.
NO CRASHES, NOT EVEN SCRATCHES.
I did this in 30 hours. 3 Days. Of which, half of the time I'm probably waiting, fucking around, or procrastinating. OR TRYING TO LOAD THE FUCKING PART SUCCESSFULLY BECAUSE IT WAS DESIGNED WITHOUT GRAVITY IN MIND – part keeps falling out of it cause it loads sideways – and the parts we have are not of best quality yet.
Here is another recent achievement of mine. A single KUKA robot cell that interfaces with Allen-Bradley PLC where my PLC guy kept asking me what the Safety Network Number is… It's not a Fanuc, stop asking for that, I can't find it on the KUKA KRC4 robot, and nor did any manual show me where it is. He eventually just figured it out. I was never asked for this for a Motoman robot either. This is a very simple cell but any robot tech that worked with a Fanuc and a KUKA will tell you it's much easier and faster to get a Fanuc running. I got it running from scratch within a week. By the start of the second week, it's running parts at 100% auto – albeit not full production.
I worked with a Coherix 3D vision scanner for dispensing with Motoman Material Handler (it didn't have proper documentation on how to interface with Motoman robots, I had to email around and figured it out later). 2 Coherix had to scan 21 meters of the bead. Coherix Camera unit literally ran out of memory trying to scan more than half of it. I made it work.
Here comes the surprise:
I get paid 20 USD/hr.
TL, DR:
The company treats me pretty well but pays me little, and my fucking ego still keeps cranking out double or triple the amount of work I'm paid for. A lot of people come and go at my company and the public opinion of my company is basically the place you go to learn all this stuff.
Either they have no fucking idea what I'm doing or have to do to make shit work, or they are exploiting me. Or they think someone else did all the work.
Guess what I know? As my project manager said. Lots of people just fucks around getting paid more than I do.